Resource allocation in distributed processing systems

ABSTRACT

A distributed processing system is disclosed herein. The distributed processing system includes a server, a database server, and an application server that are interconnected via a network, and connected via the network to a plurality of independent processing units. The independent processing units can include an analysis engine that is machine-learning-capable, and thus uniquely completes its processing tasks. The server can provide one or several pieces of data to one or several of the independent processing units, can receive analysis results from these one or several independent processing units, and can update the result based on a value characterizing the machine learning of the independent processing unit.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of Ser. No. 15/824,960, filed Nov.28, 2017, and entitled “RESOURCE ALLOCATION IN DISTRIBUTED PROCESSINGSYSTEMS,” which is a divisional of U.S. application Ser. No. 14/869,748,filed on Sep. 29, 2015, and entitled “RESOURCE ALLOCATION IN DISTRIBUTEDPROCESSING SYSTEMS,” which claims the benefit of U.S. ProvisionalApplication No. 62/057,140, filed on Sep. 29, 2014, and entitled“RESOURCE ALLOCATION IN DISTRIBUTED PROCESSING SYSTEMS,” the entirety ofeach which are hereby incorporated by reference herein.

BACKGROUND

A distributed system is a software system in which components located onnetworked computers communicate and coordinate their actions by passingmessages. The components interact with each other in order to achieve acommon goal.

Use of a distributed system to perform distributed processing providesmany benefits in that distributed processing can efficiently combine andutilize processing resources at a variety of locations to quickly solvelarge and/or complicated problems. In one example of this, a group ofnetworked computers may be controlled by a group of users. However, asnormal use of a computer does not always use the computer's entireprocessing capacity, each of the computers in this group of computersmay have some latent or unused processing capacity. This latent orunused processing capacity can be identified and combined together tosolve another problem. Thus, use of the processing capacity of thecomputers of the group of computers is maximized, which maximization canlead to significant infrastructure savings associated with acquisitionof otherwise needed processing capacity.

However, distributed processing still has many shortcomings which limitsits usefulness. Thus, new methods, techniques, and systems are requiredto improve distributed processing.

BRIEF SUMMARY

One aspect of the present disclosure relates to a distributed processingnetwork that can improve processing speeds. The network includes asource device that can provide groups of data, each of which groups ofdata can be associated with one or several user authors, and whichgroups of data together can include a processing task. The networkincludes a plurality of independent processing units that can receive aportion of the processing task, which portion of the processing task caninclude one or several of the groups of data, and which independentprocessing units can characterize one or several aspects of the one orseveral of the groups of data. The network can include a servercommunicatively connected to the source device and the plurality ofindependent processing units via a network. In some embodiments, theserver can receive a signal encoding the processing task, identify aplurality of features in some of the groups of data, generate apreliminary subset from the groups of data of the processing task,calculate a subset measure for the preliminary subset, which subsetmeasure indicates the degree to which the subset is representative ofthe entire processing task, maximize the subset measure by replacingsome of the data groups of the subset, and provide a final subset to theplurality of independent processing units.

In some embodiments, the server can receive a characterization of thegroups of data of the final subset. In some embodiments, generating thepreliminary subset includes determining the desired size of thepreliminary subset. In some embodiments, the server can generate atleast one selection attribute for at least some of the groups of data inthe preliminary subset.

In some embodiments, the server can generate an attribute vector for atleast some of the groups of data of the preliminary subset. In someembodiments, the attribute vector is generated from values indicative ofthe identification of one or several of the selection attributes in theselected piece of data. In some embodiments, the attribute vectorincludes a multi-dimensional vector, and the dimensions of the attributevector correspond with selection attributes such that each dimension ofthe attribute vector is associated with a unique one of the selectionattributes.

In some embodiments, maximizing the subset measure includes calculatingcontribution factors for each of the groups of data included in thesubset. In some embodiments, each of the contribution factors identifiesthe effect of the associated group of data on the subset measure. Insome embodiments, maximizing the subset measure includes calculatingcontribution factors for some of the groups of data not included in thesubset.

In some embodiments, maximizing the subset measure includes: identifyingthe group of data in the subset having a first contribution factorindicating the smallest positive effect on the subset measure, andidentifying the group of data outside of the subset having a secondcontribution factor indicating the largest positive effect on the subsetmeasure. In some embodiments, maximizing the subset measure includes:comparing the first contribution factor and the second contributionfactor, and replacing the group of data in the subset having a firstcontribution factor indicating the smallest positive effect on thesubset measure with the group of data outside of the subset having asecond contribution factor indicating the largest positive effect on thesubset measure when the second contribution factor indicates a greaterpositive effect than the first contribution factor. In some embodiments,maximizing the subset measure includes identifying the subset asmaximized when the second contribution factor indicates a lesserpositive effect than the first contribution factor.

One aspect of the present disclosure relates to a distributed processingnetwork that can improve processing speeds. The network can include asource device that can provide groups of data, each of which groups ofdata can be associated with one or several user authors, and whichgroups of data together can form a processing task. The network caninclude a plurality of independent processing units that can receive aportion of the processing task, which portion of the processing task caninclude one or several of the groups of data, and which independentprocessing units can characterize one or several aspects of the one orseveral of the groups of data. The network can include a servercommunicatively connected to the source device and the plurality ofindependent processing units via a network. In some embodiments, theserver can receive a signal encoding the processing task, identify aplurality of features in some of the groups of data, generate anattribute vector for each of some of the groups of data in theprocessing task, which attribute vector can include a dimension relatingto the plurality of features, and which attribute vector can include amulti-dimensional vector, determine the distance between the ends of theattribute vectors, generate a subset including the pair of attributevectors having ends separated by the greatest distance, and provide thesubset to the plurality of independent processing units.

In some embodiments, the server can receive a characterization of thegroups of data of the final subset. In some embodiments, generating thepreliminary subset includes determining the desired size of thepreliminary subset. In some embodiments, the server can generate atleast one selection attribute for at least some of the groups of data inthe preliminary subset.

In some embodiments, the server can generate an attribute vector for atleast some of the groups of data of the preliminary subset. In someembodiments, the attribute vector is generated from values indicative ofthe identification of one or several of the selection attributes in theselected piece of data. In some embodiments, the attribute vectorincludes a multi-dimensional vector, and the dimensions of the attributevector correspond with selection attributes such that each dimension ofthe attribute vector is associated with a unique one of the selectionattributes.

Further areas of applicability of the present disclosure will becomeapparent from the detailed description provided hereinafter. It shouldbe understood that the detailed description and specific examples, whileindicating various embodiments, are intended for purposes ofillustration only and are not intended to necessarily limit the scope ofthe disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is described in conjunction with the appendedfigures:

FIG. 1 is a block diagram showing illustrating an example of a dataextraction and analysis system.

FIG. 2 is a block diagram illustrating a computer server and computingenvironment within a data extraction and analysis system.

FIG. 3 is a block diagram illustrating an embodiment of one or moredatabase servers within a data extraction and analysis system.

FIG. 4 is a block diagram illustrating an embodiment of one or morecontent management servers within a data extraction and analysis system.

FIG. 5 is a block diagram illustrating the physical and logicalcomponents of a special-purpose computer device within a data extractionand analysis system.

FIG. 6 is a schematic illustration of one embodiment of an independentprocessing unit of a distributed processing system.

FIG. 7 is a schematic illustration of one embodiment of the server ofone embodiment of a distributed processing system.

FIG. 8 is a swim-lane diagram illustrating one embodiment of a processfor increasing the efficiency and processing speed of distributedcomputing.

FIG. 9 is a flowchart illustrating one embodiment of a process forcalculating an actual score with a distributed processing system.

FIG. 10 is a flowchart illustrating one embodiment of a process forcalculating an accuracy parameter with a distributed processing system.

FIG. 11 is a flowchart illustrating one embodiment of a process foranalyzing a piece of data and/or a group of data with a distributedprocessing system.

FIG. 12 is a flowchart illustrating one embodiment of a process forgenerating the evaluation with a distributed processing system.

FIG. 13 a flowchart illustrating one embodiment of a process forparallelized evaluation with a distributed processing system.

FIG. 14 a flowchart illustrating one embodiment of a process foranalyzing a piece of data and/or a group of data with a distributedprocessing system.

FIG. 15 a flowchart illustrating one embodiment of a process forgenerating the relative ranking with a distributed processing system.

FIG. 16 is a flowchart illustrating one embodiment of a process forselecting a subset.

FIG. 17 is a flowchart illustrating one embodiment of a process forselecting a final sub set.

FIG. 18 is a flowchart illustrating one embodiment of a process formaximizing a measure indicative of a value of the subset.

FIG. 19 is a flowchart illustrating an alternative embodiment of aprocess for selecting a subset.

In the appended figures, similar components and/or features may have thesame reference label. Where the reference label is used in thespecification, the description is applicable to any one of the similarcomponents having the same reference label. Further, various componentsof the same type may be distinguished by following the reference labelby a dash and a second label that distinguishes among the similarcomponents. If only the first reference label is used in thespecification, the description is applicable to any one of the similarcomponents having the same first reference label irrespective of thesecond reference label.

DETAILED DESCRIPTION

The present disclosure relates to systems, methods, and devices forincreasing the efficiency and processing speed of distributed computing.Particularly, in some embodiments, distributed computing can be used toevaluate and/or analyze a plurality of unique pieces of data or datagroups. In some embodiments the number of pieces of data or data groupsmay be large, and the processing times for some or all of the pieces ofdata and/or data groups can be long. Because of this, significantprocessing resources can be required for completion of a processingtask, which can include a plurality of the pieces of data and/or datagroups.

In some embodiments, the processing can be performed by one or severalindependent processing units which can, in some embodiments, beintelligent in that they are able to learn and adapt based on externalstimulus. Particularly, the one or several independent processing unitscan learn and thereby improve the quality of their output as theyevaluate one or several of the pieces of data. While there are manybenefits of such a system of intelligent independent processing units,such a system is disadvantaged in that the evaluation of the independentprocessing unit is unique as it is based on the learning, which in manycases is also unique, of that independent processing unit. Further, inmany instances, the uniqueness of the evaluation of the independentprocessing unit is based on so many variables, many of which areunknown, that it is difficult to identify the source of the uniqueness,and thereby seek to eliminate the uniqueness.

Instead of attempting to identify a source of uniqueness, in someembodiments, the uniqueness of the evaluations can be eliminated and/ormitigated through one or several standardization processes and systems.

In one embodiment, these processes and systems can utilize one orseveral independent processing units to provide multiple evaluations forthe same piece of data and/or data group. These evaluations can becombined, and in some embodiments, combined according to an accuracyparameter that is associated with one or several independent processingunits, to generate a combined evaluation. This accuracy parameter can bebased on data relating to the degree of difference between one or moreprevious evaluations of one or more pieces of data and/or data groups byone or more independent processing units and one or more finalevaluations for each of the one or more pieces of data and/or datagroups.

In another embodiment, the processes and systems can simplify theprocessing task to mitigate the adverse effect of uniqueness on theevaluation of one or more pieces of data and/or data groups. In such anembodiment, pieces of data and/or data groups can be associated in, forexample, groups of 2 (pairs), groups of 3, groups of 4, groups of 5,groups of 10, and/or any other or intermediate value. These associationsof pieces of data and/or groups of data can be provided to one orseveral independent processing units, which independent processing unitscan relatively rank the pieces of data and/or groups of data from bestto worst. The relative rankings can then be used to assign a finalevaluation to one or several of the pieces of data and/or groups ofdata.

Advantageously, both of the above-mentioned embodiments increase theeffective processing speed with which the processing task is completed,additionally, as both of these mentioned embodiments decrease the totalnumber of evaluations of each of the pieces of data and/or groups ofdata used to generate a final evaluation. Additionally, by identifyingthe accuracy parameter, the distributed processing system is able todirect the pieces of data and/or groups of data to the independentprocessor best suited for generation of the evaluation.

With reference now to FIG. 1, a block diagram is shown illustratingvarious components of a content delivery network 100, also referred toherein as a distributed processing system 100, which implements andsupports certain embodiments and features described herein. The contentdelivery network 100 may include one or more content management servers102. As discussed below in more detail, content management servers 102may be any desired type of server including, for example, a rack server,a tower server, a miniature server, a blade server, a mini rack server,a mobile server, an ultra-dense server, a super server, or the like, andmay include various hardware components, for example, a motherboard, aprocessing units, memory systems, hard drives, network interfaces, powersupplies, etc. Content management server 102 may include one or moreserver farms, clusters, or any other appropriate arrangement and/orcombination or computer servers. Content management server 102 may actaccording to stored instructions located in a memory subsystem of theserver 102, and may run an operating system, including any commerciallyavailable server operating system and/or any other operating systemsdiscussed herein.

The content delivery network 100 may include one or more databasesservers 104, also referred to herein as databases. The database servers104 can access data that can be stored on a variety of hardwarecomponents. These hardware components can include, for example,components forming tier 0 storage, components forming tier 1 storage,components forming tier 2 storage, and/or any other tier of storage. Insome embodiments, tier 0 storage refers to storage that is the fastesttier of storage in the database server 104, and particularly, the tier 0storage is the fastest storage that is not RAM or cache memory. In someembodiments, the tier 0 memory can be embodied in solid state memorysuch as, for example, a solid-state drive (SSD) and/or flash memory.

In some embodiments, the tier 1 storage refers to storage that is one orseveral higher performing systems in the memory management system, andthat is relatively slower than tier 0 memory, and relatively faster thanother tiers of memory. The tier 1 memory can be one or several harddisks that can be, for example, high-performance hard disks. These harddisks can be one or both of physically or communicatively connected suchas, for example, by one or several fiber channels. In some embodiments,the one or several disks can be arranged into a disk storage system, andspecifically can be arranged into an enterprise class disk storagesystem. The disk storage system can include any desired level ofredundancy to protect data stored therein, and in one embodiment, thedisk storage system can be made with grid architecture that createsparallelism for uniform allocation of system resources and balanced datadistribution.

In some embodiments, the tier 2 storage refers to storage that includesone or several relatively lower performing systems in the memorymanagement system, as compared to the tier 0 and tier 1 storages. Thus,tier 2 memory is relatively slower than tier 1 and tier 0 memories. Tier2 memory can include one or several SATA-drives or one or severalNL-SATA drives.

In some embodiments, the one or several hardware and/or softwarecomponents of the database server 104 can be arranged into one orseveral storage area networks (SAN), which one or several storage areanetworks can be one or several dedicated networks that provide access todata storage, and particularly that provides access to consolidated,block level data storage. A SAN typically has its own network of storagedevices that are generally not accessible through the local area network(LAN) by other devices. The SAN allows access to these devices in amanner such that these devices appear to be locally attached to the userdevice.

Databases 104 may comprise stored data relevant to the functions of thecontent delivery network 100. Illustrative examples of databases 104that may be maintained in certain embodiments of the content deliverynetwork 100 are described below in reference to FIG. 3. In someembodiments, multiple databases may reside on a single database server104, either using the same storage components of server 104 or usingdifferent physical storage components to assure data security andintegrity between databases. In other embodiments, each database mayhave a separate dedicated database server 104.

The content delivery network 100 also may include one or more userdevices 106 and/or supervisor devices 110. User devices 106 andsupervisor devices 110 may display content received via the contentdelivery network 100, and may support various types of user interactionswith the content. In some embodiments, the user devices 106 and thesupervisor devices 110 can be configured to access data in, edit datain, retrieve data from, and/or provide data to the data extraction andanalysis system.

User devices 106 and supervisor devices 110 may include mobile devicessuch as smartphones, tablet computers, personal digital assistants, andwearable computing devices. Such mobile devices may run a variety ofmobile operating systems, and may be enabled for Internet, e-mail, shortmessage service (SMS), Bluetooth®, mobile radio-frequency identification(M-RFID), and/or other communication protocols. Other user devices 106and supervisor devices 110 may be general purpose personal computers orspecial-purpose computing devices including, by way of example, personalcomputers, laptop computers, workstation computers, projection devices,and interactive room display systems. Additionally, user devices 106 andsupervisor devices 110 may be any other electronic devices, such asthin-client computers, Internet-enabled gaming system, business or homeappliances, and/or personal messaging devices, capable of communicatingover network(s) 120. In some embodiments, the designated role of adevice, including a user device 106 or a supervisor device 110 can varybased on the identity of the user using that device. Thus, in someembodiments, both user and supervisor devices 106, 110 can include thesame hardware, but can be configured as one of a user device 106 or asupervisor device 110 at the time of log-in by a user to use thatdevice.

In different contexts of data extraction and analysis systems 100, userdevices 106 and supervisor devices 110 may correspond to different typesof specialized devices, for example, student devices and teacher devicesin an educational network, employee devices and presentation devices ina company network, different gaming devices in a gaming network, etc. Insome embodiments, user devices 106 and supervisor devices 110 mayoperate in the same physical location, such as a classroom, a clinic, orconference room. In such cases, the devices may contain components thatsupport direct communications with other nearby devices, such as awireless transceivers and wireless communications interfaces, Ethernetsockets or other Local Area Network (LAN) interfaces, etc. In otherimplementations, the user devices 106 and supervisor devices 110 neednot be used at the same location 107, but may be used in remotegeographic locations in which each user device 106 and supervisor device110 may use security features and/or specialized hardware (e.g.,hardware-accelerated SSL and HTTPS, WS-Security, firewalls, etc.) tocommunicate with the content management server 102 and/or other remotelylocated user devices 106. Additionally, different user devices 106 andsupervisor devices 110 may be assigned different designated roles, suchas presenter devices, teacher devices, administrator devices, or thelike, and in such cases the different devices may be provided withadditional hardware and/or software components to provide content andsupport user capabilities not available to the other devices.

The distributed processing system 100 can include one or severalindependent processing units 113. In some embodiments, the independentprocessing units 113 can each be a computing device comprising aprocessor, which can comprise a microprocessor, such as a microprocessorfrom Intel® or Advanced Micro Devices, Inc.®, or the like, memory, auser interface configured to provide information to a user and toreceive one or several inputs from a user, or the like. In someembodiments, the independent processing units 113 can be computers.

In some embodiments, the independent processing units 113 can beconfigured to receive one or several data pieces and/or groups of datafrom the server 102 or other component of the distributed processingsystem 100 and to evaluate the one or several data pieces and/or groupsof data. In some embodiments, this evaluation can include a relativeranking of the one or more pieces of data and/or the groups of data. Insome embodiments, this evaluation can include a characterization of thepieces of data and/or groups of data according to one or severalcriteria and/or guidelines. In some embodiments, the independentprocessing units 113 can be intelligent in that they are able to learnand adapt based on external stimulus. Particularly, the one or severalindependent processing units 113 can learn and thereby improve thequality of their output as they evaluate one or several of the pieces ofdata and/or groups of data. Thus, in some embodiments, the evaluationsprovided by the independent processing units 113 vary based on thelearning of the independent processing units 113. In some embodiments,and in contrast to the independent processing units 113, the server 102is not able to learn and adapt based on external stimulus. Thus,evaluations provided by the server 102 are performed according toinstructions received by the server and are consistent unless thoseinstructions change.

The content delivery network 100 also may include a privacy server 108that maintains private user information at the privacy server 108 whileusing applications or services hosted on other servers. For example, theprivacy server 108 may be used to maintain private data of a user withinone jurisdiction even though the user is accessing an application hostedon a server (e.g., the content management server 102) located outsidethe jurisdiction. In such cases, the privacy server 108 may interceptcommunications between a user device 106 or supervisor device 110 andother devices that include private user information. The privacy server108 may create a token or identifier that does not disclose the privateinformation and may use the token or identifier when communicating withthe other servers and systems, instead of using the user's privateinformation.

As illustrated in FIG. 1, the content management server 102 may be incommunication with one or more additional servers, such as a contentserver 112, a user data server 114, and/or an administrator server 116.Each of these servers may include some or all of the same physical andlogical components as the content management server(s) 102, and in somecases, the hardware and software components of these servers 112-116 maybe incorporated into the content management server(s) 102, rather thanbeing implemented as separate computer servers.

Content server 112 may include hardware and software components togenerate, store, and maintain the content resources for distribution touser devices 106 and other devices in the network 100. For example, indata extraction and analysis systems 100 used for professional trainingand educational purposes, content server 112 may include databases oftraining materials, presentations, plans, syllabi, reviews, evaluations,interactive programs and simulations, course models, course outlines,and various training interfaces that correspond to different materialsand/or different types of user devices 106. In data extraction andanalysis systems 100 used for media distribution, interactive gaming,and the like, a content server 112 may include media content files suchas music, movies, television programming, games, and advertisements. Insome embodiments, the one or several content servers 112 can be incommunicating connection with the content management server 102 via, forexample, the internet or other communication network.

In some embodiments, the content delivery network 100 can include aplurality of content servers 112 that can contain the same or differentcontent. In some embodiments, this plurality of content servers 112 canbe controlled as a part of the content delivery network 100, and in someembodiments, this plurality of content servers 112 can be controlledindependent of the content delivery network 100. In such an embodiments,data can be transferred to and/or from one or several of the pluralityof content servers 112 and some or all of the other components of thecontent delivery network 100.

In one embodiment, for example, the content delivery network 100 caninclude a first content server, a second content server, a third contentserver, and/or a fourth content server. In some embodiments, forexample, some or all of the first, second, third, and fourth contentservers can host websites, which can be unique. These websites cancontain information that can be retrieved and/or used by some or all ofthe other components of the content delivery network 100. In someembodiments, the first content server can be configured to host and/orcan host a first website containing a first portion of species data, thesecond content server can be configured to host and/or can host a secondportion of species data, the third content server can be configured tohost and/or can host a first portion of qualitative data, and/or thefourth content server can be configured to host and/or can host a secondportion of qualitative data.

User data server 114 may include hardware and software components thatstore and process data for multiple users relating to each user'sactivities and usage of the content delivery network 100. For example,the content management server 102 may record and track each user'ssystem usage, including their user device 106, content resourcesaccessed, and interactions with other user devices 106. This data may bestored and processed by the user data server 114, to support usertracking and analysis features. For instance, in the professionaltraining and educational contexts, the user data server 114 may storeand analyze each user's training materials viewed, presentationsattended, courses completed, interactions, evaluation results, and thelike. The user data server 114 may also include a repository foruser-generated material, such as evaluations and tests completed byusers, and documents and assignments prepared by users. In the contextof media distribution and interactive gaming, the user data server 114may store and process resource access data for multiple users (e.g.,content titles accessed, access times, data usage amounts, gaminghistories, user devices and device types, etc.).

Administrator server 116 may include hardware and software components toinitiate various administrative functions at the content managementserver 102 and other components within the content delivery network 100.For example, the administrator server 116 may monitor device status andperformance for the various servers, databases, and/or user devices 106in the content delivery network 100. When necessary, the administratorserver 116 may add or remove devices from the network 100, and performdevice maintenance such as providing software updates to the devices inthe network 100. Various administrative tools on the administratorserver 116 may allow authorized users to set user access permissions tovarious content resources, monitor resource usage by users and devices106, and perform analyses and generate reports on specific network usersand/or devices (e.g., resource usage tracking reports, trainingevaluations, etc.).

The content delivery network 100 may include one or more communicationnetworks 120. Although only a single network 120 is identified in FIG.1, the content delivery network 100 may include any number of differentcommunication networks between any of the computer servers and devicesshown in FIG. 1 and/or other devices described herein. Communicationnetworks 120 may enable communication between the various computingdevices, servers, and other components of the content delivery network100. As discussed below, various implementations of data extraction andanalysis systems 100 may employ different types of networks 120, forexample, computer networks, telecommunications networks, wirelessnetworks, and/or any combination of these and/or other networks.

In some embodiments, some of the components of the content deliverynetwork 100 can belong to the content network 122. The content network122 can include, for example, the content management server 102, thedatabase server 104, the privacy server 108, the content server 112, theuser data server 114, the administrator server 116, and/or thecommunication network 120. The content network 122 can be the source ofcontent distributed by the content delivery network 100, which contentcan include, for example, one or several documents and/or applicationsor programs. These documents and/or applications or programs are digitalcontent. In some embodiments, these one or several documents and/orapplications or programs can include, for example, one or severalwebpages, presentations, papers, videos, charts, graphs, books, writtenwork, figures, images, graphics, recordings, applets, scripts, or thelike.

As used herein, an “evaluation” refers to the result of an analysis ofone or several pieces of data and/or groups of data that includes both acharacterization and an explanation of the characterization. As usedherein, a “relative ranking” refers to the identification of therelative quality of one or several of the one or several pieces of dataand/or groups of data as compared to each other. As used herein, a“characterization” refers to an indicator of the result of theevaluation. This indicator can be a numeric value, an alphabeticcharacter, an alphanumeric character, any other symbol, or the like.

As used herein, the “explanation of the characterization” can identifyone or several factors leading to the characterization. These factorscan be identified in analysis criteria that identify one or severalanalysis parameters, the degree to which they are triggered, and/or thefrequency with which they are triggered in the one or several pieces ofdata and/or groups of data determine a parameter value. In someembodiments, the aggregate of the parameter values of analysis criteriadetermines the characterization of the one or several pieces of dataand/or groups of data. The analysis parameters can, in some embodiments,describe one or several attributes of the one or several pieces of dataand/or data groups. In some embodiments, these one or several of theseattributes can be computer detectable. In some embodiments, the degreeto which the attributes occur, and/or the frequency with which theyoccur influences the degree to which, and/or the frequency with whichthe analysis parameters are triggered, and thereby influences theparameter value.

In some embodiments, the analysis criteria can be unique and/or specificto each processing task comprising a plurality of pieces of data and/orgroups of data, and in some embodiments, the analysis criteria can benon-unique to each processing task. The analysis criteria can be storedin memory of the content delivery network 100 such as, for example, theserver 102 and/or the database server 104. In some embodiments, theanalysis criteria can be received from the content server 112; in someembodiments, the analysis criteria can be received from a user device106 connected with the server 102; and, in some embodiments, theanalysis criteria can be directly input into the server 102.

In some embodiments, the analysis criteria can be applied according toan analysis guideline. In some embodiments, the analysis guidelineidentifies one or several rules for application of the analysiscriteria. The analysis guideline can increase the uniformity acrossevaluations. In some embodiments, the analysis guideline can be uniqueand/or specific to each processing task comprising a plurality of piecesof data and/or groups of data, and in some embodiments, the analysisguideline can be non-unique to each processing task. The analysisguideline can be stored in the memory of the content delivery network100 such as, for example, the server 102 and/or the database server 104.In some embodiments, the analysis guideline can be received from thecontent server 112; in some embodiments, the analysis guideline can bereceived from a user device 106 connected with the server 102; and, insome embodiments, the analysis guideline can be directly input into theserver 102.

With reference to FIG. 2, an illustrative distributed computingenvironment 200 is shown including a computer server 202, four clientcomputing devices 206, and other components that may implement certainembodiments and features described herein. In some embodiments, theserver 202 may correspond to the content management server 102 discussedabove in FIG. 1, and the client computing devices 206 may correspond tothe user devices 106. However, the computing environment 200 illustratedin FIG. 2 may correspond to any other combination of devices and serversconfigured to implement a client-server model or other distributedcomputing architecture.

Client devices 206 may be configured to receive and execute clientapplications over one or more networks 220. Such client applications maybe web browser based applications and/or standalone softwareapplications, such as mobile device applications. Server 202 may becommunicatively coupled with the client devices 206 via one or morecommunication networks 220. Client devices 206 may receive clientapplications from server 202 or from other application providers (e.g.,public or private application stores). Server 202 may be configured torun one or more server software applications or services, for example,web-based or cloud-based services, to support content distribution andinteraction with client devices 206. Users operating client devices 206may in turn utilize one or more client applications (e.g., virtualclient applications) to interact with server 202 to utilize the servicesprovided by these components.

Various different subsystems and/or components 204 may be implemented onserver 202. Users operating the client devices 206 may initiate one ormore client applications to use services provided by these subsystemsand components. The subsystems and components within the server 202 andclient devices 206 may be implemented in hardware, firmware, software,or combinations thereof. Various different system configurations arepossible in different distributed computing systems 200 and dataextraction and analysis systems 100. The embodiment shown in FIG. 2 isthus one example of a distributed computing system and is not intendedto be limiting.

Although exemplary computing environment 200 is shown with four clientcomputing devices 206, any number of client computing devices may besupported. Other devices, such as specialized sensor devices, etc., mayinteract with client devices 206 and/or server 202.

As shown in FIG. 2, various security and integration components 208 maybe used to send and manage communications between the server 202 anduser devices 206 over one or more communication networks 220. Thesecurity and integration components 208 may include separate servers,such as web servers and/or authentication servers, and/or specializednetworking components, such as firewalls, routers, gateways, loadbalancers, and the like. In some cases, the security and integrationcomponents 208 may correspond to a set of dedicated hardware and/orsoftware operating at the same physical location and under the controlof same entities as server 202. For example, components 208 may includeone or more dedicated web servers and network hardware in a datacenteror a cloud infrastructure. In other examples, the security andintegration components 208 may correspond to separate hardware andsoftware components which may be operated at a separate physicallocation and/or by a separate entity.

Security and integration components 208 may implement various securityfeatures for data transmission and storage, such as authenticating usersand restricting access to unknown or unauthorized users. In variousimplementations, security and integration components 208 may provide,for example, a file-based integration scheme or a service-basedintegration scheme for transmitting data between the various devices inthe content delivery network 100. Security and integration components208 also may use secure data transmission protocols and/or encryptionfor data transfers, for example, File Transfer Protocol (FTP), SecureFile Transfer Protocol (SFTP), and/or Pretty Good Privacy (PGP)encryption.

In some embodiments, one or more web services may be implemented withinthe security and integration components 208 and/or elsewhere within thecontent delivery network 100. Such web services, including cross-domainand/or cross-platform web services, may be developed for enterprise usein accordance with various web service standards, such as the WebService Interoperability (WS-I) guidelines. For example, some webservices may use the Secure Sockets Layer (SSL) or Transport LayerSecurity (TLS) protocol to provide secure connections between the server202 and user devices 206. SSL or TLS may use HTTP or HTTPS to provideauthentication and confidentiality. In other examples, web services maybe implemented using the WS-Security standard, which provides for secureSOAP messages using XML, encryption. In other examples, the security andintegration components 208 may include specialized hardware forproviding secure web services. For example, security and integrationcomponents 208 may include secure network appliances having built-infeatures such as hardware-accelerated SSL and HTTPS, WS-Security, andfirewalls. Such specialized hardware may be installed and configured infront of any web servers, so that any external devices may communicatedirectly with the specialized hardware.

Communication network(s) 220 may be any type of network familiar tothose skilled in the art that can support data communications using anyof a variety of commercially-available protocols, including withoutlimitation, TCP/IP (transmission control protocol/Internet protocol),SNA (systems network architecture), IPX (Internet packet exchange),Secure Sockets Layer (SSL) or Transport Layer Security (TLS) protocols,Hyper Text Transfer Protocol (HTTP) and Secure Hyper Text TransferProtocol (HTTPS), and the like. Merely by way of example, network(s) 220may be local area networks (LAN), such as one based on Ethernet,Token-Ring and/or the like. Network(s) 220 also may be wide-areanetworks, such as the Internet. Networks 220 may includetelecommunication networks such as a public switched telephone networks(PSTNs), or virtual networks such as an intranet or an extranet.Infrared and wireless networks (e.g., using the Institute of Electricaland Electronics (IEEE) 802.11 protocol suite or other wirelessprotocols) also may be included in networks 220.

Computing environment 200 also may include one or more databases 210and/or back-end servers 212. In certain examples, the databases 210 maycorrespond to database server(s) 104 discussed above in FIG. 1, andback-end servers 212 may correspond to the various back-end servers112-116. Databases 210 and servers 212 may reside in the same datacenteror may operate at a remote location from server 202. In some cases, oneor more databases 210 may reside on a non-transitory storage mediumwithin the server 202. Other databases 210 and back-end servers 212 maybe remote from server 202 and configured to communicate with server 202via one or more networks 220. In certain embodiments, databases 210 andback-end servers 212 may reside in a storage-area network (SAN). In someembodiments, the computing environment can be replicated for each of thenetworks 122 and 104 discussed with respect to FIG. 1 above.

With reference to FIG. 3, an illustrative set of databases and/ordatabase servers is shown, corresponding to the databases servers 104 ofthe content delivery network 100 discussed above in FIG. 1. One or moreindividual databases 301-310 may reside in storage on a single computerserver 104 (or a single server farm or cluster) under the control of asingle entity, or may reside on separate servers operated by differententities and/or at remote locations. In some embodiments, databases301-310 may be accessed by the content management server 102 and/orother devices and servers within the network 100 (e.g., user devices106, supervisor devices 110, administrator servers 116, etc.). Access toone or more of the databases 301-310 may be limited or denied based onthe processes, user credentials, and/or devices attempting to interactwith the database.

The paragraphs below describe examples of specific databases that may beimplemented within some embodiments of a content delivery network 100.It should be understood that the below descriptions of databases301-310, including their functionality and types of data stored therein,are illustrative and non-limiting. Database server architecture, design,and the execution of specific databases 301-310 may depend on thecontext, size, and functional requirements of a content delivery network100. For example, in content distribution systems 100 used forprofessional training and educational purposes, separate databases maybe implemented in database server(s) 104 to store trainee and/or studentdata, trainer and/or professor data, training module data and contentdescriptions, training results, evaluation data, and the like. Incontrast, in content distribution systems 100 used for mediadistribution from content providers to subscribers, separate databasesmay be implemented in database server(s) 104 to store listing ofavailable content titles and descriptions, content title usagestatistics, subscriber profiles, account data, payment data, networkusage statistics, etc.

A user profile database 301 may include information relating to the endusers within the content delivery network 100. Generally speaking theuser profile database 301 can be a database having restrictions onaccess, which restrictions can relate to whether one or several users orcategories of users are enabled to perform one or several actions on thedatabase or on data stored in the database. In some embodiments, theuser profile database 301 can include any information for which accessis restricted. This information may include user characteristics such asthe user names, access credentials (e.g., logins and passwords), userpreferences, and information relating to any previous user interactionswithin the content delivery network 100 (e.g., requested content, postedcontent, content modules completed, training scores or evaluations,other associated users, etc.). In some embodiments, this information canrelate to one or several individual end users such as, for example, oneor several students, teachers, administrators, or the like, and in someembodiments, this information can relate to one or several institutionalend users such as, for example, one or several schools, groups ofschools such as one or several school districts, one or severalcolleges, one or several universities, one or several trainingproviders, or the like.

In some embodiments in which the one or several end users areindividuals, and specifically are students, the user profile database301 can further include information relating to these students' academicand/or educational history. This information can identify one or severalcourses of study that the student has initiated, completed, and/orpartially completed, as well as grades received in those courses ofstudy. In some embodiments, the student's academic and/or educationalhistory can further include information identifying student performanceon one or several tests, quizzes, and/or assignments. In someembodiments, this information can be stored in a tier of memory that isnot the fastest memory in the content delivery network 100.

The user profile database 301 can include information relating to one orseveral student learning preferences. In some embodiments, for example,the student may have one or several preferred learning styles, one orseveral most effective learning styles, and/or the like. In someembodiments, the students learning style can be any learning styledescribing how the student best learns or how the student prefers tolearn. In one embodiment, these learning styles can include, forexample, identification of the student as an auditory learner, as avisual learner, and/or as a tactile learner. In some embodiments, thedata identifying one or several student learning styles can include dataidentifying a learning style based on the student's educational historysuch as, for example, identifying a student as an auditory learner whenthe student has received significantly higher grades and/or scores onassignments and/or in courses favorable to auditory learners. In someembodiments, this information can be stored in a tier of memory that isnot the fastest memory in the content delivery network 100.

The user profile database 301 can further include information relatingto one or several teachers and/or instructors who are responsible fororganizing, presenting, and/or managing the presentation of informationto the student. In some embodiments, user profile database 301 caninclude information identifying courses and/or subjects that have beentaught by the teacher, data identifying courses and/or subjectscurrently taught by the teacher, and/or data identifying courses and/orsubjects that will be taught by the teacher. In some embodiments, thiscan include information relating to one or several teaching styles ofone or several teachers. In some embodiments, the user profile database301 can further include information indicating past evaluations and/orevaluation reports received by the teacher. In some embodiments, theuser profile database 301 can further include information relating toimprovement suggestions received by the teacher, training received bythe teacher, continuing education received by the teacher, and/or thelike. In some embodiments, this information can be stored in a tier ofmemory that is not the fastest memory in the content delivery network100.

An accounts database 302 may generate and store account data fordifferent users in various roles within the content delivery network100. For example, accounts may be created in an accounts database 302for individual end users, supervisors, administrator users, and entitiessuch as companies or educational institutions. Account data may includeaccount types, current account status, account characteristics, and anyparameters, limits, restrictions associated with the accounts.

A content library database 303 may include information describing theindividual content items (or content resources) available via thecontent delivery network 100. In some embodiments, the library database303 may include metadata, properties, and other characteristicsassociated with the content resources stored in the content server 112.In some embodiments, this data can include the one or several items thatcan include one or several documents and/or one or several applicationsor programs. In some embodiments, the one or several items can include,for example, one or several webpages, presentations, papers, videos,charts, graphs, books, written work, figures, images, graphics,recordings, or any other document, or any desired software orapplication or component thereof including, for example, a graphicaluser interface (GUI), all or portions of a Learning Management System(LMS), all or portions of a Content Management System (CMS), all orportions of a Student Information Systems (SIS), or the like.

In some embodiments, the content library database 303 can include aplurality of content items for presentation in one or severalvalidations. These content items, also referred to herein as questions,can be any desired type of question including, for example, multiplechoice questions, true/false questions, essay questions, short-answerquestions, oral questions, demonstration questions, or the like. Some orall of the content items can be associated with data that can be used insorting the content items, in selecting one or several content items,and/or in the evaluation of one or several content items. This data caninclude, for example, data identifying topics relevant to the contentitems. These topics can identify, for example, competencies demonstratedby the student in correctly responding to and/or answering thequestions, and/or potentially lacking competencies demonstrated by thestudent in incorrectly responding to and/or answering the content items.In some embodiments, a content item can be associated with a singletopic, and in some embodiments, a content item can be associated withmultiple topics. Thus, in some embodiments, the correct answering of asingle content item may be indicative of a number of competencies, andthe incorrect answering of a single content item may be indicative of anumber of incompetencies. In some embodiments, the content librarydatabase 303 can further include a database of topics. This database oftopics can include some or all of the topics associated with contentitems. In some embodiments, the content library database 303 can beconfigured such that content items associated with a topic can beprovided in response to a query for the same, and likewise such thattopics associated with a content item can be provided in response to aquery for the same.

In some embodiments, the data in the content library database 303 mayidentify one or more aspects or content attributes of the associatedcontent resources, for example, subject matter, access level, or skilllevel of the content resources, license attributes of the contentresources (e.g., any limitations and/or restrictions on the licensableuse and/or distribution of the content resource), price attributes ofthe content resources (e.g., a price and/or price structure fordetermining a payment amount for use or distribution of the contentresource), rating attributes for the content resources (e.g., dataindicating the evaluation or effectiveness of the content resource), andthe like. In some embodiments, the library database 303 may beconfigured to allow updating of content metadata or properties, and toallow the addition and/or removal of information relating to the contentresources. In some embodiments, the content library database 303 can beorganized such that content is associated with one or several coursesand/or programs in which the content is used and/or provided. In someembodiments, the content library database 303 can further include one orseveral teaching materials used in the course, a syllabus, one orseveral practice problems, one or several tests, one or several quizzes,one or several assignments, or the like. All or portions of the contentlibrary database can be stored in a tier of memory that is not thefastest memory in the content delivery network 100.

A pricing database 304 may include pricing information and/or pricingstructures for determining payment amounts for providing access to thecontent delivery network 100 and/or the individual content resourceswithin the network 100. In some cases, pricing may be determined basedon a user's access to the content delivery network 100, for example, atime-based subscription fee, or pricing based on network usage, or thelike. In other cases, pricing may be tied to specific content resources.Certain content resources may have associated pricing information,whereas other pricing determinations may be based on the resourcesaccessed, the profiles and/or accounts of the users and the desiredlevel of access (e.g., duration of access, network speed, etc.).Additionally, the pricing database 304 may include information relatingto compilation pricing for groups of content resources, such as groupprices and/or price structures for groupings of resources.

A license database 305 may include information relating to licensesand/or licensing of the content resources within the content deliverynetwork 100. For example, the license database 305 may identify licensesand licensing terms for individual content resources and/or compilationsof content resources in the content server 112, the rights holders forthe content resources, and/or common or large-scale right holderinformation such as contact information for rights holders of contentnot included in the content server 112.

A content access database 306 may include access rights and securityinformation for the content delivery network 100 and specific contentresources. For example, the content access database 306 may includelogin information (e.g., user identifiers, logins, passwords, etc.) thatcan be verified during user login attempts to the network 100. Thecontent access database 306 also may be used to store assigned rolesand/or levels of access to users. For example, a user's access level maycorrespond to the sets of content resources and/or the client or serverapplications that the user is permitted to access. Certain users may bepermitted or denied access to certain applications and resources basedon their subscription level, training program, course/grade level, etc.Certain users may have supervisory access over one or more end users,allowing the supervisor to access all or portions of the end user'scontent, activities, evaluations, etc. Additionally, certain users mayhave administrative access over some users and/or some applications inthe content management network 100, allowing such users to add andremove user accounts, modify user access permissions, performmaintenance updates on software and servers, etc.

A source database 307 can comprise information relating to the source ofsome or all of the one or several pieces of data and/or groups of dataincluding, for example, the one or several originating devices. In someembodiments, the source database 307 can include information identifyingthe originating device and/or the user of the originating device, theauthor of some or all of the one or several pieces of data and/or datagroups, previous pieces of data and/or groups of data originating fromthe originating device and/or the author, or the like. In someembodiments, the information stored in the source database can bereceived via the content server 112.

An evaluation database 308 can include information used to direct theevaluation of the pieces of data and/or groups of data. In someembodiments, the evaluation database 308 can contain, for example, theanalysis criteria and the analysis guideline. This information can bereceived from the content server 112, and ultimately from one or severaloriginating devices.

The evaluation database 308 can further include information relating toone or several processing tasks. This information can include, forexample, the identification of independent processing units 113 thathave received one or several pieces of data and/or groups of data forevaluation, the status of the evaluation of one or several pieces ofdata and/or data groups that have been received by an independentprocessing unit 113, an accuracy parameter for one or several of theindependent processing units 113, an evaluation history for the one orseveral independent processing units 113, or the like. In someembodiments, the evaluation criteria can comprise data and/orinstructions in the form of one or several electronic rubrics or scoringguides for use in the evaluation of the one or several pieces of dataand/or data groups. The information stored within the evaluationdatabase 308 can be generated within the distributed processing system100, and, in some embodiments, this information can be provided to thedistributed processing system 100.

In some embodiments, the evaluation database 308 can include one orseveral evaluations for one or several pieces of data and/or groups ofdata. In some embodiments, these evaluations can include a relativeranking, a characterization, and/or an explanation of thecharacterization and/or relative ranking.

The distributed processing system 100 can include a control database309. The control database 309 can include information used in themanagement of the one or several independent processing units 113. Insome embodiments, this can, for example, include information used inevaluating the quality of the evaluations of the independent processingunits 113, information and/or computer code for generating an accuracyparameter, one or several pieces of learning data and/or learning groupsof data that can provide one or several external stimuli to facilitatelearning of the analysis engine 254, or the like. In some embodiments,these one or several pieces of learning data and/or learning groups ofdata can comprise training materials that are associated with a knowncharacterization and, in some embodiments, can exhibit certain traitsand/or contain certain predetermined material. This training materialcan allow the determination of the quality of the evaluations providedby the one or several independent processing units 113, and particularlyprovided by the analysis modules 204 of the one or several independentprocessing units 113.

In addition to the illustrative databases described above, databaseserver(s) 104 may include one or more external data aggregators 310.External data aggregators 310 may include third-party data sourcesaccessible to the content management network 100, but not maintained bythe content management network 100. External data aggregators 310 mayinclude any electronic information source relating to the users, contentresources, or applications of the content delivery network 100. Forexample, external data aggregators 310 may be third-party databasescontaining demographic data, education related data, consumer salesdata, health related data, and the like. Illustrative external dataaggregators 310 may include, for example, social networking web servers,public records databases, learning management systems, educationalinstitution servers, business servers, consumer sales databases, medicalrecord databases, etc. Data retrieved from various external dataaggregators 310 may be used to verify and update user accountinformation, suggest user content, and perform user and contentevaluations.

With reference now to FIG. 4, a block diagram is shown illustrating anembodiment of one or more content management servers 102 within acontent delivery network 100. As discussed above, content managementserver(s) 102 may include various server hardware and softwarecomponents that manage the content resources within the content deliverynetwork 100 and provide interactive and adaptive content to users onvarious user devices 106. For example, content management server(s) 102may provide instructions to and receive information from the otherdevices within the content delivery network 100, in order to manage andtransmit content resources, user data, and server or client applicationsexecuting within the network 100.

A content management server 102 may include a content customizationsystem 402. The content customization system 402 may be implementedusing dedicated hardware within the content delivery network 100 (e.g.,a content customization server 402), or using designated hardware andsoftware resources within a shared content management server 102. Insome embodiments, the content customization system 402 may adjust theselection and adaptive capabilities of content resources to match theneeds and desires of the users receiving the content. For example, thecontent customization system 402 may query various databases and servers104 to retrieve user information, such as user preferences andcharacteristics (e.g., from a user profile database 301), user accessrestrictions to content recourses (e.g., from a content access database306), and the like. Based on the retrieved information from databases104 and other data sources, the content customization system 402 maymodify content resources for individual users.

A content management server 102 also may include a user managementsystem 404. The user management system 404 may be implemented usingdedicated hardware within the content delivery network 100 (e.g., a usermanagement server 404), or using designated hardware and softwareresources within a shared content management server 102. In someembodiments, the user management system 404 may monitor the progress ofusers through various types of content resources and groups, such asmedia compilations, courses or curriculums in training or educationalcontexts, interactive gaming environments, and the like. For example,the user management system 404 may query one or more databases andservers 104 to retrieve user data such as associated contentcompilations or programs, content completion status, user goals,results, and the like.

A content management server 102 also may include an evaluation system406. The evaluation system 406 may be implemented using dedicatedhardware within the content delivery network 100 (e.g., an evaluationserver 406), or using designated hardware and software resources withina shared content management server 102. The evaluation system 406 may beconfigured to receive and analyze information from user devices 106 via,for example, the privacy server 108. For example, various ratings ofcontent resources submitted by users may be compiled and analyzed, andthen stored in a database (e.g., a content library database 303)associated with the content. In some embodiments, the evaluation server406 may analyze the information to determine the effectiveness orappropriateness of content resources with, for example, a subjectmatter, an age group, a skill level, or the like. In some embodiments,the evaluation system 406 may provide updates to the contentcustomization system 402 or the user management system 404, with theattributes of one or more content resources or groups of resourceswithin the network 100. The evaluation system 406 also may receive andanalyze user evaluation data from user devices 106, supervisor devices110, and administrator servers 116, etc. For instance, evaluation system406 may receive, aggregate, and analyze user evaluation data fordifferent types of users (e.g., end users, supervisors, administrators,etc.) in different contexts (e.g., media consumer ratings, trainee orstudent comprehension levels, teacher effectiveness levels, gamer skilllevels, etc.).

A content management server 102 also may include a content deliverysystem 408. The content delivery system 408 may be implemented usingdedicated hardware within the content delivery network 100 (e.g., acontent delivery server 408), or using designated hardware and softwareresources within a shared content management server 102. The contentdelivery system 408 may receive content resources from the contentcustomization system 402 and/or from the user management system 404, andprovide the resources to user devices 106. The content delivery system408 may determine the appropriate presentation format for the contentresources based on the user characteristics and preferences, and/or thedevice capabilities of user devices 106. If needed, the content deliverysystem 408 may convert the content resources to the appropriatepresentation format and/or compress the content before transmission. Insome embodiments, the content delivery system 408 may also determine theappropriate transmission media and communication protocols fortransmission of the content resources.

In some embodiments, the content delivery system 408 may includespecialized security and integration hardware 410, along withcorresponding software components to implement the appropriate securityfeatures content transmission and storage, to provide the supportednetwork and client access models, and to support the performance andscalability requirements of the network 100. The security andintegration layer 410 may include some or all of the security andintegration components 208 discussed above in FIG. 2, and may controlthe transmission of content resources and other data, as well as thereceipt of requests and content interactions, to and from the userdevices 106, supervisor devices 110, administrative servers 116, andother devices in the network 100.

With reference now to FIG. 5, a block diagram of an illustrativecomputer system is shown. The system 500 may correspond to any of thecomputing devices or servers of the content delivery network 100described above, or any other computing devices described herein. Inthis example, computer system 500 includes processing units 504 thatcommunicate with a number of peripheral subsystems via a bus subsystem502. These peripheral subsystems include, for example, a storagesubsystem 510, an I/O subsystem 526, and a communications subsystem 532.

Bus subsystem 502 provides a mechanism for letting the variouscomponents and subsystems of computer system 500 communicate with eachother as intended. Although bus subsystem 502 is shown schematically asa single bus, alternative embodiments of the bus subsystem may utilizemultiple buses. Bus subsystem 502 may be any of several types of busstructures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures. Sucharchitectures may include, for example, an Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnect (PCI) bus, which can beimplemented as a Mezzanine bus manufactured to the IEEE P1386.1standard.

Processing unit 504, which may be implemented as one or more integratedcircuits (e.g., a conventional microprocessor or microcontroller),controls the operation of computer system 500. One or more processors,including single core and/or multicore processors, may be included inprocessing unit 504. As shown in the figure, processing unit 504 may beimplemented as one or more independent processing units 506 and/or 508with single or multicore processors and processor caches included ineach processing unit. In other embodiments, processing unit 504 may alsobe implemented as a quad-core processing unit or larger multicoredesigns (e.g., hexa-core processors, octo-core processors, ten-coreprocessors, or greater.

Processing unit 504 may execute a variety of software processes embodiedin program code, and may maintain multiple concurrently executingprograms or processes. At any given time, some or all of the programcode to be executed can be resident in processor(s) 504 and/or instorage subsystem 510. In some embodiments, computer system 500 mayinclude one or more specialized processors, such as digital signalprocessors (DSPs), outboard processors, graphics processors,application-specific processors, and/or the like.

I/O subsystem 526 may include device controllers 528 for one or moreuser interface input devices and/or user interface output devices 530.User interface input and output devices 530 may be integral with thecomputer system 500 (e.g., integrated audio/video systems, and/ortouchscreen displays), or may be separate peripheral devices which areattachable/detachable from the computer system 500.

Input devices 530 may include a keyboard, pointing devices such as amouse or trackball, a touchpad or touch screen incorporated into adisplay, a scroll wheel, a click wheel, a dial, a button, a switch, akeypad, audio input devices with voice command recognition systems,microphones, and other types of input devices. Input devices 530 mayalso include three dimensional (3D) mice, joysticks or pointing sticks,gamepads and graphic tablets, and audio/visual devices such as speakers,digital cameras, digital camcorders, portable media players, webcams,image scanners, fingerprint scanners, barcode reader 3D scanners, 3Dprinters, laser rangefinders, and eye gaze tracking devices. Additionalinput devices 530 may include, for example, motion sensing and/orgesture recognition devices that enable users to control and interactwith an input device through a natural user interface using gestures andspoken commands, eye gesture recognition devices that detect eyeactivity from users and transform the eye gestures as input into aninput device, voice recognition sensing devices that enable users tointeract with voice recognition systems through voice commands, medicalimaging input devices, MIDI keyboards, digital musical instruments, andthe like.

Output devices 530 may include one or more display subsystems, indicatorlights, or non-visual displays such as audio output devices, etc.Display subsystems may include, for example, cathode ray tube (CRT)displays, flat-panel devices, such as those using a liquid crystaldisplay (LCD) or plasma display, projection devices, touch screens, andthe like. In general, use of the term “output device” is intended toinclude all possible types of devices and mechanisms for outputtinginformation from computer system 500 to a user or other computer. Forexample, output devices 530 may include, without limitation, a varietyof display devices that visually convey text, graphics and audio/videoinformation such as monitors, printers, speakers, headphones, automotivenavigation systems, plotters, voice output devices, and modems.

Computer system 500 may comprise one or more storage subsystems 510,comprising hardware and software components used for storing data andprogram instructions, such as system memory 518 and computer-readablestorage media 516. The system memory 518 and/or computer-readablestorage media 516 may store program instructions that are loadable andexecutable on processing units 504, as well as data generated during theexecution of these programs.

Depending on the configuration and type of computer system 500, systemmemory 518 may be stored in volatile memory (such as random accessmemory (RAM) 512) and/or in non-volatile storage drives 514 (such asread-only memory (ROM), flash memory, etc.) The RAM 512 may contain dataand/or program modules that are immediately accessible to and/orpresently being operated and executed by processing units 504. In someimplementations, system memory 518 may include multiple different typesof memory, such as static random access memory (SRAM) or dynamic randomaccess memory (DRAM). In some implementations, a basic input/outputsystem (BIOS), containing the basic routines that help to transferinformation between elements within computer system 500, such as duringstart-up, may typically be stored in the non-volatile storage drives514. By way of example, and not limitation, system memory 518 mayinclude application programs 520, such as client applications, Webbrowsers, mid-tier applications, server applications, etc., program data522, and an operating system 524.

Storage subsystem 510 also may provide one or more tangiblecomputer-readable storage media 516 for storing the basic programmingand data constructs that provide the functionality of some embodiments.Software (programs, code modules, instructions) that when executed by aprocessor provide the functionality described herein may be stored instorage subsystem 510. These software modules or instructions may beexecuted by processing units 504. Storage subsystem 510 may also providea repository for storing data used in accordance with the presentinvention.

Storage subsystem 510 may also include a computer-readable storage mediareader that can further be connected to computer-readable storage media516. Together and, optionally, in combination with system memory 518,computer-readable storage media 516 may comprehensively representremote, local, fixed, and/or removable storage devices plus storagemedia for temporarily and/or more permanently containing, storing,transmitting, and retrieving computer-readable information.

Computer-readable storage media 516 containing program code, or portionsof program code, may include any appropriate media known or used in theart, including storage media and communication media, such as, but notlimited to, volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information. This can include tangible computer-readable storagemedia such as RAM, ROM, electronically erasable programmable ROM(EEPROM), flash memory or other memory technology, CD-ROM, digitalversatile disk (DVD), or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or other tangible computer readable media. This can also includenontangible computer-readable media, such as data signals, datatransmissions, or any other medium which can be used to transmit thedesired information and which can be accessed by computer system 500.

By way of example, computer-readable storage media 516 may include ahard disk drive that reads from or writes to non-removable, nonvolatilemagnetic media, a magnetic disk drive that reads from or writes to aremovable, nonvolatile magnetic disk, and an optical disk drive thatreads from or writes to a removable, nonvolatile optical disk such as aCD ROM, DVD, Blu-Ray® disk, or other optical media. Computer-readablestorage media 516 may include, but is not limited to, Zip® drives, flashmemory cards, universal serial bus (USB) flash drives, secure digital(SD) cards, DVD disks, digital video tape, and the like.Computer-readable storage media 516 may also include, solid-state drives(SSD) based on non-volatile memory such as flash-memory based SSDs,enterprise flash drives, solid state ROM, and the like, SSDs based onvolatile memory such as solid state RAM, dynamic RAM, static RAM,DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs thatuse a combination of DRAM and flash memory based SSDs. The disk drivesand their associated computer-readable media may provide non-volatilestorage of computer-readable instructions, data structures, programmodules, and other data for computer system 500.

Communications subsystem 532 may provide a communication interface fromcomputer system 500 and external computing devices via one or morecommunication networks, including local area networks (LANs), wide areanetworks (WANs) (e.g., the Internet), and various wirelesstelecommunications networks. As illustrated in FIG. 5, thecommunications subsystem 532 may include, for example, one or morenetwork interface controllers (NICs) 534, such as Ethernet cards,Asynchronous Transfer Mode NICs, Token Ring NICs, and the like, as wellas one or more wireless communications interfaces 536, such as wirelessnetwork interface controllers (WNICs), wireless network adapters, andthe like. Additionally and/or alternatively, the communicationssubsystem 532 may include one or more modems (telephone, satellite,cable, ISDN), synchronous or asynchronous digital subscriber line (DSL)units, FireWire® interfaces, USB® interfaces, and the like.Communications subsystem 536 also may include radio frequency (RF)transceiver components for accessing wireless voice and/or data networks(e.g., using cellular telephone technology, advanced data networktechnology, such as 3G, 4G or EDGE (enhanced data rates for globalevolution), WiFi (IEEE 802.11 family standards, or other mobilecommunication technologies, or any combination thereof), globalpositioning system (GPS) receiver components, and/or other components.

The various physical components of the communications subsystem 532 maybe detachable components coupled to the computer system 500 via acomputer network, a FireWire® bus, or the like, and/or may be physicallyintegrated onto a motherboard of the computer system 500. Communicationssubsystem 532 also may be implemented in whole or in part by software.

In some embodiments, communications subsystem 532 may also receive inputcommunication in the form of structured and/or unstructured data feeds,event streams, event updates, and the like, on behalf of one or moreusers who may use or access computer system 500. For example,communications subsystem 532 may be configured to receive data feeds inreal-time from users of social networks and/or other communicationservices, web feeds such as Rich Site Summary (RSS) feeds, and/orreal-time updates from one or more third party information sources(e.g., data aggregators 310). Additionally, communications subsystem 532may be configured to receive data in the form of continuous datastreams, which may include event streams of real-time events and/orevent updates (e.g., sensor data applications, financial tickers,network performance measuring tools, clickstream analysis tools,automobile traffic monitoring, etc.). Communications subsystem 532 mayoutput such structured and/or unstructured data feeds, event streams,event updates, and the like to one or more databases 104 that may be incommunication with one or more streaming data source computers coupledto computer system 500.

Due to the ever-changing nature of computers and networks, thedescription of computer system 500 depicted in the figure is intendedonly as a specific example. Many other configurations having more orfewer components than the system depicted in the figure are possible.For example, customized hardware might also be used and/or particularelements might be implemented in hardware, firmware, software, or acombination. Further, connection to other computing devices, such asnetwork input/output devices, may be employed. Based on the disclosureand teachings provided herein, a person of ordinary skill in the artwill appreciate other ways and/or methods to implement the variousembodiments.

With reference now to FIG. 6, a schematic illustration of one embodimentof the independent processing unit 113 is shown. As discussed above, theindependent processing unit 113 can be configured to provide informationto and/or receive information from other components of the distributedprocessing system 100. The independent processing unit 113 can accessthe distributed processing system 100 through any desired means ortechnology, including, for example, a webpage, a web portal, or vianetwork 120. As depicted in FIG. 6, the independent processing unit 113can include a network interface 250. The network interface 250 allowsthe independent processing unit 113 to access the other components ofthe distributed processing system 100, and specifically allows theindependent processing unit 113 to access the network 120 of thedistributed processing system 100. The network interface 250 can includeone or several features configured to send and receive information,including, for example, an antenna, a modem, a transmitter, receiver, orany other feature that can send and receive information. The networkinterface 250 can communicate via telephone, cable, fiber-optic, or anyother wired communication network. In some embodiments, the networkinterface 250 can communicate via cellular networks, WLAN networks, orany other wireless network.

The independent processing unit 113 can include a user interface 252that communicates information to, and receives inputs from a user. Theuser interface 252 can include a screen having one or several pixels, aspeaker, a monitor, a keyboard, a microphone, a mouse, a touchpad, akeypad, or any other feature or features that can receive inputs from auser and provide information to a user. In some embodiments, the pixelsof the screen can be controlled to provide information to the user ofthe independent processing unit 113. In some embodiments, the userinterface 252 can allow the user to provide one or several inputs to theindependent processing unit 113 that are relevant to the othercomponents of the distributed processing system 100 discussed herein.

The independent processing unit 113 can include an analysis engine 254.In some embodiments, the analysis engine 254 can be configured toreceive one or several pieces of data and/or groups of data from theserver 102, and/or from the database 104. In some embodiments, theanalysis engine 254 can be an embodiment in hardware or software thatcan be configured to retrieve the analysis criteria and the analysisguideline. In some embodiments, the analysis engine 254 can beconfigured to learn and adapt based on external stimulus, andparticularly based on the one or several of the pieces of data or datagroups previously evaluated by the analysis engine 254. In someembodiments, the analysis engine can be configured to generate a userinterface that can facilitate the human review of the one or severalpieces of data and/or data groups.

The analysis engine 254 can generate the evaluation according to theanalysis criteria and the analysis guideline. In such an embodiment, theanalysis module 254 can identify one or several analysis parameters ofthe analysis criteria, can determine the degree to which the attributesassociated with the analysis parameters occur, and/or the frequency withwhich the attributes occur in the one or several pieces of data and/orgroups of data. These attributes can relate to, for example, word usage,vocabulary, spelling, grammar, content, citations, use of citations, orthe like. In one embodiment, for example, these criteria can include thenumber and/or frequency of misspellings, the number and/or frequency ofgrammar errors, the level of vocabulary of the one or several pieces ofdata and/or data groups, the correctness and/or level of word usage, thecoherency, accuracy, and/or relevancy of the pieces of data and/or datagroups, or the like.

The independent processing unit 113 can include a standards engine 256,also referred to herein as a standards module. The standards engine 256can be embodied in hardware or software. In some embodiments, thestandards engine 256 can be configured to receive and store the analysiscriteria and the analysis guideline. In some embodiments, the standardsengine 256 can receive the analysis criteria and the analysis guidelinefrom another component of the distributed processing system 100 such as,for example, the server 102 and/or the database server 106. Thestandards engine 256 can be further configured to maintain the operationof the analysis module 254 in accordance with the analysis criteria andthe analysis guideline. In some embodiments, the standards engine 256can be configured to convert all or portions of the analysis criteriaand/or analysis guidelines to visual format and to control the userinterface 252 to display these. In some embodiments, the standardsengine 256 can be configured to convert the analysis criteria and/oranalysis guideline from the form in which they were provided to thedistributed processing system 100 to one or several lines of computercode that can control the operation of the analysis engine 254.

With reference now to FIG. 7, a schematic illustration of one embodimentof the server 102 is shown. The server 102 includes a network interface450. The network interface 450 allows the server 102 to access the othercomponents of the distributed processing system 100, and specificallyallows the server 102 to access the network 120 of the distributedprocessing system 100. The server 102 can include features configured tosend and receive information, including, for example, an antenna, amodem, a transmitter, receiver, or any other feature that can send andreceive information. The network interface 450 can communicate viatelephone, cable, fiber-optic, or any other wired communication network.In some embodiments, the network interface 450 can communicate viacellular networks, WLAN networks, or any other wireless network.

The server 102 can include a user interface 452 that communicatesinformation to, and receives inputs from a user. The user interface 452can include a screen having one or several pixels, a speaker, a monitor,a keyboard, a microphone, a mouse, a touchpad, a keypad, or any otherfeature or features that can receive inputs from a user and provideinformation to a user. In some embodiments, the pixels of the screen canbe controlled to provide information to the user of the server 102. Insome embodiments, the user interface 452 can allow the user to provideone or several inputs to the server that are relevant to the othercomponents of the distributed processing system 100 discussed herein.

The server 102 can include a distribution engine 454. The distributionengine 454 can be configured to manage the distribution of theprocessing task(s) to the one or several independent processing units113. In one embodiment, the distribution engine can be configured toreceive a processing task comprising one or several pieces of dataand/or data groups from the content server 112, to identify informationrelating to the desired completion of the processing task. In someembodiments, this information can be stored within the analysisguideline, and can, for example, specify a number of evaluations forcompletion of the processing task, a number of repeated evaluations forcompletion of the processing task, one or several attributes ofindependent processing units 113 that can be used for completion of theprocessing task, which can include, for example, a language capability,a subject matter competency level, a training and/or learning level, orthe like. In some embodiments, the distribution engine 454 can receiveinformation relating to one or several of the independent processingunits 113 and can select a set of the independent processing units 113for use in completion of the processing task.

The distribution engine 454 can be further configured to monitor theprogress of the processing task, and can, for example, identify problemsin the completion of the processing task, generate commands and controlportions of the distributed processing system 100 to achieve one orseveral deadlines or to comply with one or several timelines, or thelike.

The server 102 can include a characterization engine 456. Thecharacterization engine 456 can be configured to receive one or severalevaluations from the independent processing units 113 and to generate acharacterization for some or all of the pieces of data and/or groups ofdata included in the processing task. In some embodiments, thecharacterization engine 456 can be configured to receive an accuracyparameter for some or all of the independent processing units 113 thathas generated an evaluation and to use this accuracy parameter, inconnection with the initial characterization contained in the evaluationto generate a final characterization. In some embodiments, this caninclude, for example, retrieving relative ranking information from theevaluation of the one or several pieces of data and/or data groups,generating a ranking for the pieces of data and/or groups of data in theprocessing task, and assigning one or several characterizations based onthe ranking for the pieces of data and/or groups of data in theprocessing task. The characterization engine 456 can be furtherconfigured to identify a discrepancy between the initialcharacterization of the independent processing units 113 and the finalcharacterization. This discrepancy can be compared, directly orindirectly, to the accuracy parameter of the independent processing unit113 that is the source of the initial evaluation, and this discrepancycan be used to update the accuracy parameter of that independentprocessing unit 113.

The server 102 can include a training engine 458. The training engine458 can be configured to generate and/or provide material to one orseveral independent processing units 113 to improve evaluations providedby the independent processing unit 113. In some embodiments, thetraining material can be configured to be used by the analysis engine254 for machine learning by the analysis engine 254. In someembodiments, the training material can be transmitted by the analysisengine 254 to a user of the independent processing unit 113 tofacilitate in training the user of the independent processing unit 113.The training engine 458 can be further configured to track trainingreceived by one or several of the independent processing units 113 andto generate a training plan based on, for example, the accuracyparameter, past received training, or the like.

In some embodiments, the server 102 can include an evaluation engine460. In some embodiments, the evaluation engine 460 can be configured toreceive one or several pieces of data and/or groups of data from thecontent server 112 and/or from the database 104. In some embodiments,the evaluation engine 460 can be embodiment in hardware or software thatcan be configured to retrieve the analysis criteria and the analysisguideline. In some embodiments, the analysis engine can be configured togenerate a user interface that can facilitate the human review of theone or several pieces of data and/or data groups. In contrast to theanalysis engine 254, the evaluation engine 460, in some embodiments, isnot configured to learn and adapt based on external stimulus, andparticularly based on the one or several of the pieces of data or datagroups previously evaluated by the evaluation engine 460.

The evaluation engine 460 can generate the evaluation according to theanalysis criteria and the analysis guideline. In such an embodiment, theevaluation engine 460 can identify one or several analysis parameters ofthe analysis criteria, and can determine the degree to which theattributes associated with the analysis parameters occur, and/or thefrequency with which the attributes occur in the one or several piecesof data and/or groups of data. These attributes can relate to, forexample, word usage, vocabulary, spelling, grammar, content, citations,use of citations, or the like. In one embodiment, for example, thesecriteria can include the number and/or frequency of misspellings, thenumber and/or frequency of grammar errors, the level of vocabulary ofthe one or several pieces of data and/or data groups, the correctnessand/or level of word usage, the coherency, accuracy, and/or relevancy ofthe pieces of data and/or data groups, or the like.

With reference now to FIG. 8, a swim-lane diagram illustrating oneembodiment of a process 550 for increasing the efficiency and processingspeed of distributed computing is shown. The headers of the swim lanesidentify components of the distributed processing system 100 that canperform the indicated step.

The process 550 begins in block 552 wherein the content server 112receives one or several pieces of data and/or data groups. In someembodiments, these can be received from the one or several originatingdevices. After the content server 112 receives the one or several piecesof data and/or data groups, the process 550 proceeds to block 554,wherein the one or several pieces of data and/or data groups areaggregated into a processing task. In some embodiments, this aggregationcan be performed according to information that is received with one orseveral of the one or several pieces of data and/or data groups. Thisinformation can identify a common aspect of the one or several pieces ofdata and/or data groups such as, for example, a type of the one orseveral pieces of data and/or data groups. This information relating tothe common aspect of the one or several pieces of data and/or datagroups can be used to compile the one or several pieces of data and/ordata groups into the processing task. In some embodiments, and as a partof the aggregation of the processing task, the content server 112 canreceive and/or retrieve the analysis criteria and analysis guidelinerelevant to the one or several pieces of data and/or groups of dataforming the processing task. In some embodiments, these can beaggregated into the processing task with the one or several pieces ofdata and/or groups of data. In other embodiments, however, the analysiscriteria and analysis guideline are received and/or retrieved at a laterpoint in process 550.

After the processing task has been aggregated, the process 550 proceedsto block 556, wherein the processing task is sent, from the contentserver 112 to the server 102. In some embodiments, the sending of theprocessing task includes any method or technique of moving theprocessing task from the content server 112 to the server 102. In someembodiments, this sending of the processing task can be according to apush-model, and in some embodiments, this sending can be according to apull-model. After the processing task has been sent, the process 550proceeds to block 558, wherein the server 102 receives the processingtask. In some embodiments, the server 102 can receive the processingtask via the network 120 and the network interface 450 of the server102.

After the processing task has been received by the server 102, theprocess 550 proceeds to block 560, wherein a set of independentprocessing units 113 is identified. In some embodiments, this set ofindependent processing units can be identified by evaluating theinformation received from the content server 112, including, forexample, the analysis criteria and analysis guideline. In someembodiments, this set can be identified based on the size of theprocessing task, based on any deadlines and/or timelines for completionof the processing task, or the like.

After the set of independent processing units has been identified, theprocess 550 proceeds to block 562, wherein the processing task isseparated into separate pieces of data and/or groups of data, or intogroups of pieces of data and/or groups of data. In some embodiments, theserver 102 can then generate a signal corresponding to the separatepieces of data and/or groups of data, or groups of pieces of data and/orgroups of data, and can, via signal, send the separate pieces of dataand/or groups of data, or into groups of pieces of data and/or groups ofdata to one or several of the independent processing units 113 in theset of independent processing units. In some embodiments, this includesany method or technique of moving the data from the server 102 to one orseveral of the independent processing units 113. In some embodiments,this can include sending data according to a push-model, and in someembodiments, this can include sending data according to a pull-model.

After the data has been sent, the process 550 proceeds to block 564,wherein the sent data is received by one of the independent processingunits 113. This data can be received by the independent processing unit113 via the network 120 and via the network interface 250. After thedata has been received, the process 550 proceeds to block 566, whereinthe analysis criteria and/or analysis guideline are requested. In someembodiments in which the content server 112 received the analysiscriteria and/or analysis guideline, the independent processing unit 113can request these from the content server 112. Alternatively, in someembodiments in which the content server 112 received the analysiscriteria and/or analysis guideline, these can be passed to the server102 as part of the processing task, and the server 102 can distributethese to the individual ones of the processing units 113.

In the embodiment depicted in FIG. 8, the analysis criteria and/oranalysis guideline are contained in the database 104, and theindependent processing unit 113 can request these from the database 104.In some embodiments, this can include the generation of an electronicquery by the independent processing unit 113, which query is sent to thedatabase 104. After the database 104 receives the request for theanalysis criteria and/or analysis guideline at block 568, the process550 proceeds to block 570, wherein the analysis criteria and/or analysisguideline are sent to the independent processing unit 113. In someembodiments, this can include the generation of an electrical signal,which is sent from the database 104, or the hardware housing to thedatabase 104 to the independent processing unit 113. In someembodiments, step 570 can include the receipt of the analysis criteriaand/or analysis guideline by the independent processing unit 113.

After the analysis criteria and/or analysis guideline have been sent tothe independent processing unit 113, the process 550 proceeds to block572, wherein the pieces of data and/or groups of data, and/or groups ofpieces of data received by the independent processing unit 113 at block564 are analyzed. In some embodiments, this analysis can be performedby, at, and/or with the independent processing unit, and specifically,by or with the analysis engine 254. In some embodiments, this caninclude updating of the analysis engine 254 with the analysis criteriaand analysis guideline and the use of the analysis criteria and analysisguideline to analyze the data. This can include, for example, anobjective analysis, a subject analysis, a substantive analysis, acontextual analysis, or the like. The details of some embodiments ofthis analysis will be discussed at greater lengths below.

After the data has been analyzed, the process 550 proceeds to block 574,wherein the analysis result is provided, by the independent processingunit 113 to the server 102. After the analysis result is provided, theprocess 550 proceeds to block 576, wherein the analysis results arereceived by the server 102. In some embodiments, the results can bereceived via one or several electric signals.

After the results have been received, the process 550 proceeds todecision state 578, wherein it is determined if a threshold has beenreached. In some embodiments, this threshold can identify the minimumnumber of evaluations to be performed to complete the processing task.This threshold can, in some embodiments, be specified in the analysisguideline. In some embodiments, the server 102 can track the number ofevaluations that have been performed. In one embodiment, this can bedone by incrementing a value when data is sent for evaluation and/orwhen analysis results are received. In some embodiments, this value canthen identify the number of evaluations performed for a processing task,but can also include information such as, for example, the number oftimes some or all of the one or several pieces of data and/or groups ofdata have been evaluated, which independent processing units 113evaluated which of the one or several pieces of data and/or groups ofdata, or the like.

If it is determined that the threshold has not been reached, then theprocess 550 returns to block 562, and proceeds as outlined above. If itis determined that the threshold has been reached, the process 550proceeds to block 580, wherein the accuracy parameter is received. Insome embodiments, the accuracy parameter can be received from theindependent processing unit 113 at the time that the analysis resultsare received; in some embodiments, the accuracy parameter can bereceived and/or retrieved from the database 104, and in someembodiments, the accuracy parameter can be generated by the server 102.

After the accuracy parameter has been received, the process 550 proceedsto block 582, wherein the final characterization of some or all of theone or several pieces of data and/or data groups is calculated and/orgenerated. In some embodiments, this final characterization can becalculated and/or generated with the accuracy parameter, as well as withthe evaluation results from one or several independent processing units113. In embodiments in which multiple independent processing units 113generated evaluations for a single piece of data and/or group of data,the calculation of the final characterization can include, for example,adjusting the individual characterizations according to the accuracyvalue, and combining the adjusted values to generate a finalcharacterization.

After the final characterization has been calculated, the process 550proceeds to block 584, wherein the accuracy parameter is updated. Insome embodiments, this can include comparing the final characterizationto some or all of the characterizations of the independent processingunits 113, determining a degree of difference between the two, andgenerating an instance accuracy parameter based on this difference. Insome embodiments, the instance accuracy parameter can be used to adjustthe accuracy parameter, thus if the deviation of a characterization isgreater than indicated by the accuracy parameter, the instance accuracyparameter can be combined with the accuracy parameter to update and/oradjust the accuracy parameter to better reflect this deviation.Conversely, if the deviation of a characterization is less thanindicated by the accuracy parameter, the instance accuracy parameter canbe combined with accuracy parameter to update and/or adjust the accuracyparameter to better reflect this deviation.

With reference now to FIG. 9, a flowchart illustrating one embodiment ofa process 600 for calculating an actual score is shown, which processincreases the efficiency and processing speed of a distributed computingsystem 100. In some embodiments, the process 600 can be performed by thedistributed computing system 100 and/or by a component thereofincluding, for example, the server 102, the independent processing unit113, or any other component of the distributed computing system 100.

The process begins at block 602 wherein the processing task is generatedand/or aggregated. In some embodiments, the processing task can beaggregated at the content server 112, and in some embodiments, theprocessing task can be aggregated at the server 102. In someembodiments, the aggregation of the processing task can include thecombination of one or several pieces of data and/or groups of data.

After the processing task has been generated, the process 600 proceedsto block 604 wherein independent processing unit information isretrieved and/or received. In some embodiments, this information canidentify one or several attributes of one or several independentprocessing units including, for example, one or several attributes ofthe analysis engine 254 and/or the user of the analysis engine. Theseattributes can, for example, identify the number of evaluations providedby one or several of the independent processing units 113 including, forexample, evaluations performed by the analysis engine 254, and/orevaluation performed by a user of the analysis engine 254.

After the independent processing unit information is retrieved, theprocess 600 proceeds to block 606 wherein the processing set isidentified. In some embodiments, the processing set can comprise aplurality of independent processing units and/or users of theindependent processing units. This set can be selected, in someembodiments, based on the independent processing unit information thatis retrieved in block 604, the analysis criteria, and/or analysisguideline. In some embodiments, this set can be selected by the server102 and/or by another component of the distributed processing system100.

After the processing set has been identified, the process 600 proceedsto block 608, wherein an independent processing unit 113 is selected. Insome embodiments, the selected independent processing unit 113 can beselected from the processing set. The independent processing unit can,in some embodiments, be selected based on the independent processingunit information that is retrieved in block 604, the analysis criteria,and/or analysis guideline.

After the independent processing unit 113 has been selected, the process600 proceeds to block 610 wherein one or several pieces of data and/ordata groups are encoded and/or converted into an electric signal. Insome embodiments, this can occur at the server 102 and can be part of,for example, the sending of the one or several pieces of data and/orgroups of data to the selected independent processing unit.

After the one or several pieces of data and/or data groups are encodedand/or converted into an electric signal, the process 600 proceeds toblock 612 wherein the one or several pieces of data and/or data groupsare provided. In some embodiments, this step can be performed for all ofthe one or several pieces of data and/or data groups of the processingtask in that some or all of the one or several pieces of data and/ordata groups are sent to an independent processing unit 113. In someembodiments this can result in several independent processing unitsreceiving some of the one or several pieces of data and/or data groups.

In some embodiments, the one or several pieces of data and/or datagroups are provided to the independent processing unit via, for example,the user interface 252. After the one or several pieces of data and/ordata groups have been provided, the process 600 proceeds to block 614wherein the evaluation is received. In some embodiments, the evaluationcan be received from one or several of the independent processing units113. In one embodiment, for example, in which the processing task wasdivided into a plurality of pieces of data and/or groups of data, andthen one or several of the plurality of pieces of data and/or groups ofdata was provided to a plurality of independent processing units 113,the step of block 614 can include receiving the evaluation from some orall of this plurality of independent processing units 113. In someembodiments, the evaluations can be received via, for example, thenetwork 120, and specifically via the user interface 252 of theindependent processing units 113 via the network.

After the evaluation has been received, the process 600 proceeds toblock 616 wherein the evaluation is stored. In some embodiments, forexample, the evaluation can be stored within a portion of thedistributed processing system 100 such as, for example, the database104. In one embodiment, the evaluations can be stored in one of thedatabases of the database server 104 such as, for example, theevaluation database 308.

After the evaluation has been stored, the process 600 proceeds to block618 wherein the evaluation guideline is retrieved. In some embodiments,the evaluation retrieved can be retrieved from the database 104. Theevaluation guideline can include a value indicating the number ofevaluations and/or the number of redundant evaluations to be received tocomplete the processing request. In some embodiments, a redundantevaluation can be an additional evaluation of an already evaluated pieceof data and/or group of data.

After the evaluation guideline has been retrieved, the process 600proceeds to block 620 wherein the number of completed evaluations isdetermined. In some embodiments, and as mentioned above, the number ofcompleted and/or requested evaluations can be tracked by incrementing avalue when data is sent for evaluation and/or when an evaluation isreceived. In some embodiments, this value can be specific to aprocessing task, and in some embodiments, this value can include aportion that is specific to a processing task, and a portion specific tosome or all of the pieces of data and/or data groups of the processingtask. Thus, in one embodiment, the number of times a piece of dataand/or a group of data is evaluated can be tracked, as well as the totalnumber of evaluations performed for a processing task. In someembodiments, the determination of the number of completed evaluationscan include retrieving the value tracking the number of evaluations.

After the number of completed evaluations is determined, the process 600proceeds to decision state 622 wherein it is determined if theevaluation threshold is achieved. In some embodiments, thisdetermination can be performed by the server 102, and/or anothercomponent of the distributed processing system 100. If it is determinedthat the evaluation threshold has not been achieved, then the process600 returns to block 608 and continues as outlined above.

If it is determined that the evaluation threshold has been achieved,then the process 600 proceeds to block 624 wherein characterizations areretrieved. In some embodiments, this can include retrieving the receivedevaluations, and separating the characterizations from the evaluations.In some embodiments, the evaluations can be retrieved from the database104, and the characterizations can be separated from the evaluations bythe server 102.

After the characterizations are retrieved, the process 600 proceeds toblock 626 wherein the accuracy parameter is retrieved. In someembodiments, for example, the accuracy parameter can be retrieved fromthe database 104, and specifically, from one of the databases, such as,for example, the evaluation database 308 and/or the control database 309of the database 104.

After the accuracy parameter has been retrieved, the process 600proceeds to block 628 wherein the final characterization, also referredto herein as a combined characterization, is generated. In someembodiments, for example, the combined characterization can becalculated by the server 102 of the distributed processing system 100,and can be calculated based on the retrieved characterizations for oneor several of the pieces of data and/or data groups and the accuracyparameters associated with the analysis engine 254, or user thereof, ofthe independent processing units 113 that provided the evaluation. Insome embodiments, some or all of the pieces of data and/or data groupsof the processing task can have characterizations, and a finalcharacterization based on the characterizations and the accuracyparameters associated with the analysis engine 254, or user thereof, ofthe independent processing units 113 that provided the evaluation.

After the actual score has been calculated, the process 600 proceeds toblock 630 wherein the accuracy parameter is updated. In someembodiments, for example, the updating accuracy parameter can includethe comparison of the characterization and the final characterization,the determination of the degree of deviation between thecharacterization and the final characterization, a determination of theupdate and/or augmentation to be provided to the accuracy parameter toreflect the degree of discrepancy between the characterization and thefinal characterization, and a change of the accuracy parameter based onthe update and/or augmentation.

With reference now to FIG. 10, a flowchart illustrating one embodimentof a process 700 for calculating an accuracy parameter is shown. In someembodiments, the process 700 can be performed by the distributedprocessing system 100 and/or a component thereof including, for example,the server 102.

The process 700 begins at block 702 wherein the independent processingunit 113 is identified. In some embodiments, the independent processingunit 113 can be identified as an independent processing unit 113 thatcompleted an evaluation and/or was included in the set of independentprocessing units. After the independent processing unit 113 has beenidentified, the process 700 proceeds to block 704 wherein the evaluationis retrieved. In some embodiments, the evaluation can be retrieved bycomponent of the distributed processing system 100 such as, for example,the server 102, and can be retrieved, for example, from the databaseserver 104.

After the evaluation has been retrieved, the process 700 proceeds toblock 706 wherein the characterization is retrieved. Thecharacterization can be retrieved from the evaluation retrieved in block704, and/or can be separated retrieved by, for example, the server 102from the database 104. After the characterization has been retrieved,the process 700 proceeds to block 708 wherein the final characterizationis retrieved. In some embodiments, the final characterization can beretrieved from the database 104 and/or one of the databases thereof.

After the final characterization has been retrieved, the process 700proceeds to block 710 wherein the final characterization is comparedwith the characterization. In some embodiments, this comparison can beperformed by the server 102. After the characterization and the finalcharacterization have been compared, the process 700 proceeds to block712 wherein a difference between the characterization and the finalcharacterization is determined. In some embodiments, this can includedetermining the absolute value and/or magnitude of difference betweenthe characterization and the final characterization, in someembodiments, this can include determining the degree to which thecharacterization score is higher or lower than the finalcharacterization. In some embodiments, this determination can be made bythe server 102.

After the discrepancy between the characterization and the finalcharacterization has been determined, the process 700 proceeds to block714 wherein an index function is retrieved. In some embodiments, theindex function can be used to generate an update value based on thediscrepancy and/or differences between the characterization and thefinal characterization. In some embodiments, for example, the indexfunction can be used to calculate one or several statistical parameterswith respect to one or both of the characterization and the finalcharacterization such as, for example, a standard deviation. After theindex function has been retrieved, the process 700 proceeds to block 716wherein the index function is applied. In some embodiments, for example,the index function can be applied to the server 102. In someembodiments, the application of the index function can include thedetermination of the discrepancy between the characterization and thefinal characterization in terms of the statistical parameters such as,for example, a standard deviation.

After the index function is applied, the process 700 proceeds to block718 wherein the accuracy parameter is updated. In some embodiments, forexample, the update value, which is the result of the applied indexfunction, can be used to augment and/or update the accuracy parameterthat can be, for example, stored within the database 104.

With reference now to FIG. 11, a flowchart illustrating one embodimentof a process 800 for analyzing a piece of data and/or a group of data isshown. In some embodiments, this process 800 can be performed at one orseveral of the independent processing units 113, and can be specificallyperformed by the analysis engine 254 and/or with the analysis engine 254by a user.

The process 800 begins at block 802 wherein a piece of data and/or groupof data is received. In some embodiments, this data can be received fromthe server via the network 120 and the network interface 250. After thedata has been received, the process 800 proceeds to block 804, whereinthe analysis criteria and analysis guideline are received. In someembodiments, the analysis criteria and the analysis guideline can bereceived from the server 102, from the content server 112, and/or fromthe database 104. In some embodiments, the analysis criteria andanalysis guideline can be received simultaneously with the receipt ofthe piece of data and/or the group of data, and in some embodiments, theanalysis criteria and analysis guideline can be received separately fromthe piece of data and/or the group of data. In one embodiment, theindependent processing unit 113 can generate and send a signalrequesting the analysis criteria and/or the analysis guideline, and canreceive the analysis criteria and/or analysis guideline in response tothis signal.

After the analysis criteria and analysis guideline have been received,the process 800 proceeds to block 806, wherein the analysis criteria andanalysis guideline are applied to the received piece of data and/or datagroup. In some embodiments, this step can include identifying one orseveral analysis parameters of the analysis criteria, determining thedegree to which the attributes associated with the analysis parametersoccur, and/or the frequency with which attributes associated with theanalysis parameters occur in the one or several pieces of data and/orgroups of data. These attributes can relate to, for example, word usage,vocabulary, spelling, grammar, content, citations, use of citations, orthe like. In one embodiment, for example, these criteria can include thenumber and/or frequency of misspellings, the number and/or frequency ofgrammar errors, the level of vocabulary of the one or several pieces ofdata and/or data groups, the correctness and/or level of word usage, thecoherency, accuracy, and/or relevancy of the pieces of data and/or datagroups, or the like. In some embodiments, this step can further includethe application of the analysis guideline to the received piece of dataand/or data group. In some embodiments, the application of the analysiscriteria can result in the generation of one or several parametervalues. In some embodiments, the analysis guideline can influence and/orcontrol how the analysis criteria are applied. This can include, forexample, the degree to which a parameter is evaluated and/or the levelof scrutiny applied to a parameter.

After the analysis criteria and analysis guideline have been applied tothe piece of data and/or data group, the process 800 proceeds to block808, wherein the evaluation is generated. In some embodiments, theevaluation can include information relating to the one or severalparameter values, and how the one or several parameter values werecalculated and/or determined, and in some embodiments, the evaluationcan include a characterization which can be the result of thecombination of the one or several parameter values. In some embodiments,the one or several parameter values can be combined as outlined in theanalysis criteria.

After the evaluation has been generated, the process 800 proceeds toblock 810, wherein the evaluation is provided. In some embodiments, theevaluation can be provided to the server 102 and/or any other componentof the distributed processing system 100. In some embodiments, theproviding of the evaluation can include, for example, the generation ofa signal comprising the evaluation, and the sending of the signal, viathe network 120, to the server 102 or other component of the distributedprocessing system 100.

With reference now to FIG. 12, a flowchart illustrating one embodimentof a process 900 for generating the evaluation is shown. In someembodiments, the process 900 can be performed in the place of, or as apart of one or both of blocks 806 and 808 of FIG. 11. The process 900begins at block 902, wherein an analysis parameter is selected. In someembodiments, the analysis parameter can be selected from one of theanalysis parameters of the evaluation criteria. In some embodiments, theselected analysis parameter can be a previous unselected analysisparameter.

After the analysis parameter has been selected, the process 900 proceedsto block 904, wherein one or several manifestations of the attributeassociated with the parameter are identified in the piece of data and/ordata group. In some embodiments, this can be achieved by the review ofthe piece of data and/or data group by the analysis engine and/or userof the analysis engine. The review can include, for example, a review ofspelling, grammar, substance, word usage, vocabulary level, citations,or any other attribute.

After and/or simultaneous with the identification of the one or severalmanifestations of the attribute associated with the parameter in thepiece of data and/or data group, the process 900 proceeds to block 906,wherein a parameter value associated with manifestations and/or theanalysis parameter is updated. In some embodiments, this update to thevalue can reflect the frequency of occurrence and/or the degree ofoccurrence of the attribute associated with the analysis parameter inthe piece of data and/or data group.

After the parameter value has been updated, the process 900 proceeds todecision state 908, wherein it is determined if there is an additionalparameter. In some embodiments, this can include, for example,determining whether there are any remaining, unevaluated parameters ofthe analysis criteria. If it is determined that there are additional,unevaluated parameters, then the process 900 returns to block 902 andproceeds as outlined above. If it is determined that there are notadditional, unevaluated parameters, then the process 900 proceeds toblock 910, wherein the parameter values are retrieved. In someembodiments, this can include retrieving all of the parameter values forparameters of the analysis criteria.

After the parameter values have been received, the process 900 proceedsto block 912, wherein the characterization is generated. In someembodiments, the characterization can be generated by the combination ofthe parameter values. These parameter values can be combined by theindependent processing unit 113, and/or by the analysis engine 254 oruser thereof. In some embodiments, these parameter values can becombined according to the analysis guideline.

With reference now to FIG. 13, a flowchart illustrating one embodimentof a process 1000 for parallelized evaluation is shown, whichparallelize evaluation can decrease the amount of time required toprocess an amount of pieces of data and/or data groups. In someembodiments, the process 1000 can be used to determine the relativeranking of one or several pieces of data and/or data groups within aprocessing task and/or to thereby assign and/or determine acharacterization for the one or several pieces of data and/or datagroups. In some embodiments, the process 1000 can provide for theparallel evaluation of a plurality pieces of data and/or data groups anda relative ranking of those simultaneously evaluated pieces of dataand/or data groups. The process 1000 can be performed by the distributedprocessing system 100 and/or component thereof including, for example,one or several independent processing units 113, and/or the server 102.

The process 1000 begins at block 1002 wherein two or more pieces of dataand/or data groups are collected. In some embodiments, the pieces ofdata and/or data groups can be collected by the content server 112 from,for example, one or several originating devices. Alternatively, in someembodiments, this collection can be performed by the server 102.

After the work product is collected, the process 1000 proceeds to block1004 wherein the processing task is generated and/or aggregated. In someembodiments, the processing task can be aggregated at the content server112, and in some embodiments, the processing task can be aggregated atthe server 102. In some embodiments, the aggregation of the processingtask can include the combination of one or several pieces of data and/orgroups of data.

After the processing task has been generated, the process 1000 proceedsto block 1006 wherein independent processing unit information isretrieved and/or received. In some embodiments, this information canidentify one or several attributes of one or several independentprocessing units including, for example, one or several attributes ofthe analysis engine 254 and/or the user of the analysis engine. Theseattributes can, for example, identify the number of evaluations providedby one or several of the independent processing units 113 including, forexample, evaluations performed by the analysis engine 254, and/orevaluation performed by a user of the analysis engine 254.

After the independent processing unit information is retrieved, theprocess 1000 proceeds to block 1008 wherein the processing set isidentified. In some embodiments, the processing set can comprise aplurality of independent processing units and/or users of theindependent processing units. This set can be selected, in someembodiments, based on the independent processing unit information thatis retrieved in block 1006, the analysis criteria, and/or analysisguideline. In some embodiments, this set can be selected by the server102 and/or by another component of the distributed processing system100.

After the processing set has been identified, the process 1000 proceedsto block 1010, wherein a desired number of comparisons is determined. Insome embodiments the desired number of comparisons can be defined in,for example, the analysis guideline. In some embodiments, this numbercan vary based on the degree of certainty desired for thecharacterizations and/or the amount of time, cost, and/or processingresources to be consumed by the processing task.

After the desired number of comparisons is determined, the process 1000proceeds to block 1012, wherein one or several pairs of pieces of dataand/or data groups and/or groups of pieces of data and/or data groupsare created. In some embodiments, these groups can be created so that arelative ranking and/or comparison can be performed. Thus, in oneembodiment, these groups can include at least two pieces of data and/ordata groups so that the one can be compared to the other and/orrelatively ranked with respect to the other. In some embodiments, thesepairs can be selected at random, and in some embodiments, these pairscan be selected based on some attribute of the pieces of data and/ordata groups such as, for example, the originator of the pieces of dataand/or data groups, any previous characterizations of pieces of dataand/or data groups from the originator, or the like.

After the one or several pairs of pieces of data and/or data groupsand/or groups of pieces of data and/or data groups have been generated,the process 1000 proceeds to block 1014, wherein the pairs are providedto, for example, one or several independent processing units 113. Insome embodiments, this providing can include the generation and sendingof an electric signal containing the one or several pairs of pieces ofdata and/or data groups and/or groups of pieces of data and/or datagroups. This signal can be generated and/or sent by the server 102.

After the pairs have been provided, the process 1000 proceeds to block1016, wherein the relative rank of one or several pairs of pieces ofdata and/or data groups and/or groups of pieces of data and/or datagroups are received. In some embodiments, these one or several pairs ofpieces of data and/or data groups and/or groups of pieces of data and/ordata groups can be received by the server 102 from one or several of theindependent processing units 113. These relative rankings can identifywhich of the pieces of data and/or data groups of the one or severalpairs and/or groups is (are) relatively better and/or relatively worse.After the relative rank has been received, the process 1000 proceeds toblock 1018, wherein the received relative rank information is stored. Insome embodiments, this information can be stored in, for example, acomponent of the distributed processing system 100 such as, for example,the database 104.

After the relative rank of the provided work product pieces has beenreceived, the process 1000 proceeds to decision state 1022 wherein it isdetermined if all of the pairs and/or groups in the processing task havebeen ranked. In some embodiments, this determination can be made bycomparison of the number of pairs that were sent by independentprocessing units 113 and the number of evaluations provided byindependent processing units 113. If it is determined that not allpairs/groups have been ranked, then the process can wait a predeterminedperiod and return to block 1016. Alternatively, in some embodiments, ifit is determined that an evaluation of one or several pairs/groups ismissing, the server 102 can identify the independent processing unit 113that received the pairs/groups and send a command/reminder to completethe evaluation and/or send the evaluation.

If it is determined that evaluations have been received for all of thepairs/groups, then the process 1000 proceeds to block 1024 wherein thecount is incremented. In some embodiments, the number of completedand/or requested evaluations can be tracked by incrementing a value whendata is sent for evaluation and/or when an evaluation is received. Insome embodiments, this value can be specific to a processing task, andin some embodiments, this value can include a portion that is specificto a processing task, and a portion specific to some or all of thepieces of data and/or data groups of the processing task. Thus, in oneembodiment, the number of times a piece of data and/or a group of datais evaluated can be tracked, as well as the total number of evaluationsperformed for a processing task. In some embodiments, the determinationof the number of completed evaluations can include retrieving the valuetracking the number of evaluations.

After the count has been incremented, the process 1000 proceeds to block1026 wherein the threshold is retrieved. In some embodiments, thisthreshold can identify the minimum number of evaluations to be performedto complete the processing task. This threshold can, in someembodiments, be specified in the analysis guideline and can be receivedfrom, for example, the database 104.

After the threshold has been retrieved, the process 1000 proceeds todecision state 1028 wherein it is determined if the threshold has beenachieved. In some embodiments, this can include the comparison of thevalue tracking the number of evaluations to the threshold. Thiscomparison can be performed by the server 102. In some embodiments, aBoolean-value can be assigned to the processing task indicating whetherthe threshold has been achieved. Similarly, in any embodiment describedherein, Boolean values can be assigned to indicate the outcome of adecision state, with a first Boolean value indicating a true conditioncorresponding to a “yes” answer in response to the question of thedecision state and a second Boolean value indicating a false conditioncorresponding to a “no” answer in response to the question of thedecision state. In some embodiments, the server can then use theseBoolean values to identify the outcome of the decision state, and tostore the outcome of the decision state in, for example, the database104.

If it is determined that the threshold is not achieved, then the process1000 returns to block 1010 and proceeds as outlined above. If it isdetermined that the threshold has been achieved, then the process 1000proceeds to block 1030 wherein the relative rankings of the pieces ofdata and/or data groups in the pairs/groups are compared to identifyranking discrepancies. In some embodiments, for example, the rankingdiscrepancies can be differences in the relative rankings provided bythe independent processing units 113. In some embodiments, this caninclude the retrieval of the evaluations and/or rankings from theindependent processing units 113 and/or the database 104 and thecomparison of these evaluations by the server 102.

After the ranking discrepancies have been identified, the process 1000proceeds to block 1032 wherein any identified ranking discrepancies areresolved. In some embodiments, for example, this can include determininga final relative ranking of the pieces of data and/or data groups basedon the provided rankings. In some embodiments, this can include theaveraging of the ranking, and/or the application of a weighting functionto the rankings. This resolution of the rankings of the pieces of dataand/or data groups can be performed by the server 102 and/or any othercomponent of the distributed processing system 100.

After the ranking discrepancies have been resolved, the process 1000proceeds to block 1034 wherein the processing task is ranked. In someembodiments, this can include applying the final rankings, including,for example, any resolved rankings to the processing task, andparticularly to the pieces of data and/or data groups forming theprocessing task. In some embodiments, this can be performed by theserver 102 and/or any other component of the distributed processingsystem 100.

After the processing task is ranked, the process 1000 proceeds to block1036 wherein one or several bounding data pieces are identified. In someembodiments, bounding data pieces can be pieces of data and/or groups ofdata located between characterization levels such that all the pieces ofdata and/or data groups that are relatively higher ranked than thebounding piece have a characterization from a first subset of scoresthat is higher than the characterization of the bounding piece, and allthe pieces of data and/or data groups that are ranked lower than thebounding piece have a characterization from a second subset of scoresthat is lower than the characterization of the bounding piece.

In some embodiments, the identification of the bounding pieces caninclude selecting one or several pieces of data and/or data groups ofthe processing task as potential bounding pieces, characterizing the oneor several pieces of data and/or data groups of the processing, andselecting one or several of the potential bounding pieces as boundingpieces. In some embodiments, the one or several potential boundingpieces can be selected based on one or several parameters and/orattributes of the pieces of data and/or data groups of the processingtask. In one embodiment, for example, the bounding pieces can beselected based on, for example, the past characterizations of one orseveral pieces of data and/or data groups from one or several of theoriginating device. In some embodiments, the one or several boundingpieces can be selected by performing a brief analysis of the pieces ofdata and/or data groups in the processing task.

After the potential bounding pieces have been selected, and as mentionedabove, a characterization can be given to some or all of the potentialbounding pieces. This characterization can proceed according to theprocess 900 depicted in FIG. 12. After the characterizations of thepotential bounding pieces have been received, and as mentioned above,one or several bounding pieces can be selected. In some embodiments, theone or several selected bounding pieces can, as described above, have acharacterization that is between levels.

Alternatively to the above steps for identifying one or several boundingpieces, in some embodiments, the bounding pieces can be one or severalpre-generated pieces of data and/or data groups that can be, forexample, created so as to have a desired characterization. In such anembodiment, the step of block 1036 can comprise retrieving thesebounding pieces from, for example, the database 104.

After the bounding pieces have been identified, the process 1000proceeds to block 1040 wherein the processing task is characterizedand/or in which a characterization is applied to the processing task. Insome embodiments, the characterization of the processing task can beachieved by identifying the relative rank of the one or several boundingpieces within the processing task. In some embodiments, in which thebounding pieces are pre-generated pieces of data and/or data groups,this can include performing the steps of blocks 1010 through 1018 todetermine a relative rank of the bounding pieces. In some embodiments,this application of the characterization can be performed by the server102 and/or any other component of the distributed processing system 100.

After the bounding pieces are relatively ranked, or in some embodimentsin which the bounding pieces are identified from the pieces of dataand/or groups of data of the processing task, the application of thecharacterization to the processing task can include identifying piecesof data relatively positioned between two adjacent bounding pieces, andassigning characterization based on these bounding pieces. Thus, whenthe bounding pieces are selected such that they are betweencharacterization levels, pieces of data and/or data groups that arerelatively bounded by two bounding pieces can be assigned thecharacterization level that is likewise bounded by the two boundingpieces.

In other embodiments, the characterization level can be determined basedon the separation of characterization levels between the boundingscores, and a determination of the degree of separation between theadjacent bounding scores and an intermediate piece of data and/or datagroup. This degree of separation can be expressed as a fraction having adenominator of the total number of pieces of data and/or data groupsbetween the bounding pieces, and a numerator of the number of pieces ofdata and/or data groups separating the intermediate piece of data fromone of the bounding pieces. In such an embodiment, the characterizationlevel can be calculated as the characterization level of the boundingpiece having a relatively lower characterization level plus the valuecreated by the multiplication of the fractional representation of thedegree of separation and the difference in the characterization levelsof the bounding pieces.

With reference now to FIG. 14, a flowchart illustrating one embodimentof a process 1100 for analyzing a piece of data and/or a group of datais shown. In some embodiments, this process 1100 can be performed at oneor several of the independent processing units 113, and can bespecifically performed by the analysis engine 254 and/or with theanalysis engine 254 by a user.

The process 1100 begins at block 1102 wherein a pair/group is received.In some embodiments, the pair/group can comprise two or more pieces ofdata and/or data groups. In some embodiments, this pair/group can bereceived from the server 102 via the network 120 and the networkinterface 250. After the pair/group has been received, the process 1100proceeds to block 1104, wherein the analysis criteria and analysisguideline are received. In some embodiments, the analysis criteria andthe analysis guideline can be received from the server 102, from thecontent server 112, and/or from the database 104. In some embodiments,the analysis criteria and analysis guideline can be receivedsimultaneously with the receipt of the pair/group, and in someembodiments, the analysis criteria and analysis guideline can bereceived separately from the pair/group. In one embodiment, theindependent processing unit 113 can generate and send a signalrequesting the analysis criteria and/or the analysis guideline, and canreceive the analysis criteria and/or analysis guideline in response tothis signal.

After the analysis criteria and analysis guideline have been received,the process 1100 proceeds to block 1106, wherein the analysis criteriaand analysis guideline are applied to the received pair/group. In someembodiments, this step can include identifying one or several analysisparameters of the analysis criteria, determining the relative rank ofthe pair/group with respect to these one or several analysis parameters,which can be based on, for example, the relative degree to which theattributes associated with the analysis parameters occur, and/or therelative frequency with which attributes associated with the analysisparameters occurs in the one or several pieces of data and/or groups ofdata in the pair/group. These attributes can relate to, for example,word usage, vocabulary, spelling, grammar, content, citations, use ofcitations, or the like. In one embodiment, for example, these criteriacan include the number and/or frequency of misspellings, the numberand/or frequency of grammar errors, the level of vocabulary of the oneor several pieces of data and/or data groups, the correctness and/orlevel of word usage, the coherency, accuracy, and/or relevancy of thepieces of data and/or data groups, or the like. In some embodiments,this step can further include the application of the analysis guidelineto the received pair/group. In some embodiments, the application of theanalysis criteria can result in the relative ranking of the pieces ofdata and/or data groups of the pair with respect to one or severalanalysis parameters.

After the analysis criteria and analysis guideline have been applied topair/group, the process 1100 proceeds to block 1108, wherein therelative ranking is generated. In some embodiments, the relativerankings with respect to the one or several analysis parameters can becombined to create an overall relative ranking. In some embodiments, theone or several relative rankings with respect to the analysis parameterscan be combined as outlined in the analysis criteria.

After the evaluation has been generated, the process 1100 proceeds toblock 1110, wherein the relative ranking is provided. In someembodiments, the relative ranking can be provided to the server 102and/or any other component of the distributed processing system 100. Insome embodiments, the providing of the relative ranking can include, forexample, the generation of a signal comprising the relative ranking, andthe sending of the signal, via the network 120, to the server 102 orother component of the distributed processing system 100.

With reference now to FIG. 15, a flowchart illustrating one embodimentof a process 1200 for generating the relative ranking is shown. In someembodiments, the process 1200 can be performed in the place of, or as apart of one or both of blocks 1106 and 1108 of FIG. 14. The process 1200begins at block 1202, wherein an analysis parameter is selected. In someembodiments, the analysis parameter can be selected from one of theanalysis parameters of the evaluation criteria. In some embodiments, theselected analysis parameter can be a previous, unselected analysisparameter.

After the analysis parameter has been selected, the process 1200proceeds to block 1204, wherein one or several manifestations of theattribute associated with the parameter are identified in the pieces ofdata and/or data groups of the pair/group. In some embodiments, this canbe achieved by the review of the pieces of data and/or data groups ofthe pair/group by the analysis engine 254 and/or user of the analysisengine 254. The review can include, for example, a review of spelling,grammar, substance, word usage, vocabulary level, citations, or anyother attribute.

After and/or simultaneously with the identification of the one orseveral manifestations of the attribute associated with the parameter inthe pieces of data and/or data groups of the pair/group, the process1200 proceeds to block 1206, wherein a parameter relative ranking iscreated, and particularly wherein a parameter relative ranking valueidentifying the relative degree to which the attributes associated withthe analysis parameters occur, and/or the relative frequency with whichattributes associated with the analysis parameters occurs in the one orseveral pieces of data and/or groups of data of the pair/group iscreated. In some embodiments, this can include comparing the relativedegree to which the attributes associated with the analysis parametersoccur, and/or the relative frequency with which attributes associatedwith the analysis parameters occurs in the one or several pieces of dataand/or groups of data of the pair/group, and identifying which of thepieces of data and/or data groups is relatively better with regard tothat analysis parameter. In some embodiments, a first value can beassociated with a piece of data and/or data group if it is relativelybetter with respect to an analysis parameter, a second value can beassociated with a piece of data and/or data group if it is relativelyworse with respect to an analysis parameter, and a third value can beassociated with a piece of data and/or data group if it is relativelyequal with respect to an analysis parameter.

After the parameter relative ranking has been created, the process 1200proceeds to decision state 1208, wherein it is determined if there is anadditional parameter. In some embodiments, this can include, forexample, determining whether there are any remaining, unevaluatedparameters of the analysis criteria. If it is determined that there areadditional, unevaluated parameters, then the process 1200 returns toblock 1202 and proceeds as outlined above. If it is determined thatthere are not additional, unevaluated analysis parameters, then theprocess 1200 proceeds to block 1210, wherein the parameter relativeranking values are retrieved. In some embodiments, this can includeretrieving all of the parameter relative ranking values for parametersof the analysis criteria.

After the parameter relative ranking values have been received, theprocess 1200 proceeds to block 1212, wherein the relative ranking of thepair/group is generated. In some embodiments, the relative ranking canbe generated by the combination of the parameter relative rankings. Insome embodiments, these parameter relative ranking values can becombined according to the analysis guideline. In some embodiments, forexample, some or all of the analysis parameters can be associated with aweighting value. In such an embodiment, the combined relative rankingcan be calculated by a combination of the weighting values and theparameter relative ranking values. In one embodiment, for example, someor all of the parameter ranking values can be multiplied by theirweighting value, and the products of those multiplications can be addedto create a combined value. In one such embodiment, the combined valuesof the pieces of data and/or data groups of the pair/group can becompared to determine the relative ranking of the pair/group, asindicated in block 1214 of FIG. 15. Thus, in one embodiment, a piece ofdata and/or data group can be relatively better if its combined value islarger than the combined value of another piece of data and/or datagroup. These parameter relative rankings can be generated by theindependent processing unit 113, and/or by the analysis engine 254 oruser thereof.

With reference to now FIG. 16, a flowchart illustrating one embodimentof a process 1300 for selecting a subset is shown. The process 1300 canbe performed by one or several components of the distributed processingsystem 100, and in some embodiments, the process 1300 can be performedby the server 102.

In some embodiments, the subset of the processing task can be selectedto improve the overall efficiency of use of computing resources and todecrease the total amount of time and resources used to evaluate and/oranalyze the processing task. In some embodiments, for example, thedifferent levels of evaluation, and particularly, different levels ofaccuracy of evaluation can be achieved by use of different evaluationresources. Particularly, in some embodiments, the independent processingunits 113 provide more accurate and/or more complete evaluations thanprovided by the server 102. However, the independent processing units113, in such embodiments, require more processing resources for theevaluation, more human involvement in the evaluation, and/or take moretime for generating the evaluations. Thus, use of the independentprocessing units 113 can increase costs, in resources, time, and moneyassociated with evaluations.

However, many of the benefits of evaluation of an entire processing taskby the independent processing units 113 can be achieved by selecting asubset of the processing task for evaluation by the independentprocessing units 113, and using the resulting evaluations of the subsetto improve the accuracy and quality of evaluations generated by theserver 102, which can include, for example, using the resultingevaluations of the subset to train an evaluation model that is used bythe server 102, as quality control for an evaluation model that is usedby the server 102, and/or as quality control for evaluation generated bythe server 102.

The process 1300 begins a block 1302, wherein the processing task isreceived, which processing task can include one or several pieces ofdata and/or data groups. In some embodiments, the one or several piecesof data and/or data groups can be received by the server 102 from thecontent server 112 and/or from the database 104. In some embodiments,the one or several pieces of data and/or data groups can be receivedalready organized into a single processing task, and in someembodiments, the one or several pieces of data and/or data groups can bereceived by the server 102 and then organized into the processing task.

After the processing task has been received, the process 1300 proceedsto block 1304, wherein processing task features are identified. In someembodiments, the processing task features can be the aggregate offeatures of the individual one or more pieces of data and/or datagroups. The features of the individual one or more pieces of data and/orgroups of data can be determined by evaluation of some or all of theindividual one or more pieces of data and/or groups of data. Each of thefeatures of the individual one or more pieces of data and/or groups ofdata can identify some aspect of the individual one or more pieces ofdata and/or groups of data including, for example, the size of the oneor more pieces of data and/or groups of data, including, for example,the word count, the quality of the one or more pieces of data and/orgroups of data including, for example, one or several measures ofcoherence, results of latent semantic analysis, aspects of content,punctuation, ratio of unique words to total word count, grammaticalerrors, spelling errors, or the like.

After the processing task features have been identified, the process1300 proceeds to block 1306, wherein a processing subset is identified.In some embodiments, the processing subset can be some desired number ofpieces of data and/or data groups of the processing task. This numbercan vary based on a number of parameters including, for example,available resources, acceptable costs in one or more of time, resources,and money, a desired accuracy level, or the like. The processing subsetcan be a number of pieces of data and/or groups of data that is adesired percentage of the total number of pieces of data and/or groupsof data in the processing task. This percentage can be, for example,approximately 1%, approximately 5%, approximately 10%, approximately15%, approximately 20%, approximately 25%, and/or any other orintermediate percent. As used herein, “approximately” denotes a rangethat is +/−1% of the therewith associated value, +/−2% of the therewithassociated value, +/−5% of the therewith associated value, or +/−10% ofthe therewith associated value.

After the processing subset has been selected, the process 1300 proceedsto block 1308, wherein processing is requested. In some embodiments,this can include requesting processing of the subset by, for example,one or several of the independent processing units 104, and/orrequesting processing of the one or several pieces of data and/or datagroups in the processing task and either included or not included in thesubset by, for example, the server 102.

After processing has been requested, the process 1300 proceeds to block1310, wherein the processing results are received. In some embodiments,the processing results can be received by the server 102 and can bestored in, for example, the database 104.

With reference now to FIG. 17, a flowchart illustrating one embodimentof a process 1400 for selecting a final subset is shown. The process1400 can be performed by one or several components of the distributedprocessing system 100, and in some embodiments, the process 1400 can beperformed by the server 102.

The process 1400 begins at block 1402, wherein the processing task isgenerated and/or aggregated. In some embodiments, the processing taskcan be aggregated at the content server 112, and in some embodiments,the processing task can be aggregated at the server 102. In someembodiments, the aggregation of the processing task can include thecombination of one or several pieces of data and/or groups of data.

After the processing task has been generated, the process 1400 proceedsto block 1404, wherein the number of pieces of data in the processingtask is determined. In some embodiments, for example, a count of totalnumber of pieces of data and/or groups of data in the processing taskcan be generated during the aggregation of the processing task. In suchan embodiment, this count can be retrieved and used to determine thenumber of pieces of data in the processing task. In other embodiments,after the processing task has been aggregated, the number of pieces ofdata in the processing task can be counted by, for example, the server102.

After the number of pieces of data in the processing task have beendetermined, the process 1400 proceeds to block 1406, wherein one orseveral selection attributes are determined. In some embodiments, theselection attributes correspond to the features of the one or severalpieces of data and/or groups of data discussed in reference block 1304of FIG. 16. These selection attributes identify some aspect of theindividual one or more pieces of data and/or groups of data including,for example, the size of the one or more pieces of data and/or groups ofdata, including, for example, the word count, the quality of the one ormore pieces of data and/or groups of data including, for example, one orseveral measures of coherence, results of latent semantic analysis,aspects of content, punctuation, ratio of unique words to total wordcount, grammatical errors, spelling errors, or the like. The selectionattributes can be identified by a user distributed processing system 100and can, in some embodiments, be provided by the user to the distributedprocessing system 100. In some embodiments, these provided selectionattributes can be stored in, for example, the database 104 and can bedetermined by retrieving them from the database 104.

After the selection attributes have been determined, the process 1400proceeds to block 1408, wherein one of the pieces of data and/or datagroups of the processing task is selected. After the piece of dataand/or data group has been selected, the process 1400 proceeds to block1410, wherein the selection attributes of the selected piece of data areidentified. In some embodiments, this can include analyzing the selectedpiece of data to identify data corresponding to the selectionattributes. This can include, for example, determining the size of theselected piece of data, including, for example, counting the words inthe selected piece of data, determining the quality of the selectedpiece of data which can include, for example, determining one or severalmeasures of coherence, performing latent semantic analysis on theselected piece of data, evaluating the content of the selected piece ofdata, determining the ratio of unique words to total words, identifyingthe number, type, and/or degree of spelling, grammatical, and/orpunctuation errors, or the like. In some embodiments, this can includethe association of a value indicative of the identification of one orseveral of the selection attributes in the selected piece of data withthe selected piece of data.

After the selection attributes have been identified in the selectedpiece of data, the process 1400 proceeds to block 1412, wherein anattribute vector is generated for the selected piece of data. In someembodiments, the attribute vector can be based on the values indicativeof the identification of one or several of the selection attributes inthe selected piece of data. In some embodiments, the vector can comprisea multi-dimensional vector, and the dimensions of the vector cancorrespond with selection attributes such that each dimension of thevector is associated with a unique one of the selection attributes. Insome such embodiments, the vector can have the same number of dimensionsas the number of selection attributes such that the vector can representthe combination of all of the selection attributes.

After the attribute vector has been generated, the process 1400 proceedsto block 1414, wherein the attribute vector for the selected piece ofdata is stored. In some embodiments, the attribute vector can be storedin, for example, the database 104. In some embodiments, in addition tostoring the attribute vector, the process 1400 can further includeassociating a value indicative of the completed generation of theattribute vector for the selected piece of data and/or data group, andstoring this value in, for example, the database 104.

After the attribute vector has been stored, the process 1400 proceeds toblock 1416, wherein a count is incremented. In some embodiments, thecount can track the number of pieces of data and/or data groups thathave been evaluated to identify their selection attributes. After thecount has been incremented, the process 1400 proceeds to block 1418,wherein the count is compared to the number of pieces of data and/ordata groups in the processing task. In some embodiments, this comparisoncan be performed according to a Boolean function, wherein a first value,corresponding to a “true” condition, is associated with the processingtask if the count matches the number of pieces of data in the processingtask and a second value, corresponding to a “false” condition, isassociated with the processing task if the count does not match thenumber of pieces of data in the processing tasks.

After the count has been compared with the number of pieces of data inthe processing task, the process 1400 proceeds to decision state 1422,wherein it is determined if there are any additional, un-analyzed piecesof data. In some embodiments, this determination can be made byretrieving the Boolean-value associated with the processing task, andindicative of the result of the comparison of the count to the number ofpieces of data in the processing task. If the second value is associatedwith the processing task, then there are additional pieces of dataand/or groups of data that have not been analyzed for selectionattributes, and the process returns to block 1408, wherein a piece ofdata and/or data group is selected. In some embodiments, the selectedpiece of data and/or data group does not have an associated valueindicative of completion of analysis for selection attributes and/or forthe completed generation of the attribute vector.

Returning again to decision state 1422, if the first value is associatedwith the processing task, then there are no additional pieces of dataand/or groups of data that have not been analyzed for selectionattributes, and the process 1400 proceeds to block 1424, wherein thedesired size of the processing subset is determined. In someembodiments, the desired size of the processing subset can be directlyor indirectly determined based on one or several user inputs, whichinputs may be in the form of a desired confidence level and/or accuracylevel, a desired cost level, a desired time frame for completion of theevaluation of the processing task, or the like.

After the desired size of the processing subset has been determined, theprocess 1400 proceeds to block 1426, wherein a preliminary subset isselected. The preliminary subset can include a subset of the pieces ofdata and/or data groups of the processing tasks, the number of pieces ofdata and/or data groups in which subset corresponds to the desired sizeof the processing subset. In some embodiments, the preliminary subsetcan be selected at random from the pieces of data and/or groups of dataof the processing task, and in some embodiments, the preliminary subsetcan be selected according to one or several attributes of the pieces ofdata and/or data groups of the processing task, which attributes can becontained in the attribute vectors of the pieces of data and/or datagroups.

After the preliminary subset has been selected, the process 1400proceeds to block 1430, wherein a subset measure is calculated. In someembodiments, the subset measure can identify the degree to which thepieces of data and/or groups of data are representative of the entireprocessing task and/or the degree to which the subset will create acomprehensive model of evaluations for the processing tasks. In someembodiments, this measure can be calculated using one or severaldimensions of the attribute vectors of the pieces of data and/or datagroups in the subset, and/or in some embodiments, this measure can becalculated using one or several dimensions of the attribute vectors ofthe pieces of data and/or data groups not included in the sub set.

After the subset measure has been calculated, the process 1400 proceedsto block 1432, wherein the subset measure is optimized. In someembodiments, the optimization of the subset measure can includemaximizing or minimizing the subset measure. In some embodiments, thesubset measure can be optimized by exchanging one or several of thepieces of data and/or data groups in the subset with one or several ofthe pieces of data and/or data groups not in the subset.

After the subset measure has been optimized, the process 1400 proceedsto block 1434, wherein the final subset is selected. In someembodiments, the final subset can include the pieces of data and/or datagroups that resulted in the optimized subset measure. In someembodiments, a value indicative of inclusion in the final subset can beassociated with each of the pieces of data and/or groups of dataincluded in the final subset and/or a value indicative of exclusion fromthe final subset can be associated with each of the pieces of dataand/or groups of data excluded from the final subset.

After the final subset has been identified, the process 1400 proceeds toblock 1436, wherein the final subset is provided for evaluation by, forexample, the independent processing units. In some embodiments, thisprocessing can be performed as outlined in, for example, one or severalof FIGS. 5-12 of this application. After the final subset has beenprovided, the process 1400 proceeds to block 1440, wherein acharacterization of the pieces of data in the final subset is received.In some embodiments, this characterization can be received from one orseveral of the independent processing units 113. In some embodiments,and either simultaneous with, or subsequent to one or both of blocks1436 and 1440 of FIG. 17, the process 1400 can provide the pieces ofdata and/or data groups that are not included in the subset forevaluation, which evaluation can be performed by the server 102, and theresults of this evaluation can be received and/or stored.

With reference now to FIG. 18, a flowchart illustrating one embodimentof a process 1500 for maximizing a measure indicative of a value of thesubset is shown. The process 1500 can be performed as a part of, or inplace of block 1432 of FIG. 17. The process 1500 begins at block 1502,wherein contribution factors are calculated for the pieces of data ofthe subset, and specifically, in some embodiments, a contribution factoris calculated for each of the pieces of data of the subset. In someembodiments, the contribution factor can be a measure of thecontribution of piece of data and/or group of data to the measure of thesubset. Thus, the contribution factor can indicate whether the piece ofdata and/or data group positively, negatively, and/or neutrallycontributes to the measure of the subset. In some embodiments, thecontribution factors can be calculated based, at least in part, on theattribute vectors of the pieces of data of the subset.

After the contribution factors are calculated, the process 1500 proceedsto block 1504, wherein the piece of data in the subset that makes thesmallest contribution to the optimization of the subset measure isidentified. In some embodiments, this piece of data and/or data groupcan be identified with the contribution factors of the pieces of dataand/or data groups of the subset. In embodiments in which the subsetmeasure is being maximized, the piece of data and/or data group with thesmallest contribution is the piece of data and/or data group having thesmallest contribution factor. In some embodiments, a value can beassociated with the piece of data in the subset making the smallestcontribution to identify the piece of data and/or data group as such.

After the piece of data in the subset making the smallest contributionis identified, the process 1500 proceeds to block 1506, identifying thepiece of data and/or group of data outside of the subset that would makethe largest contribution to the optimization of the subset measure. Insome embodiments, this piece of data and/or data group can be identifiedwith the contribution factors of the pieces of data and/or data groupsnot included in the subset. In embodiments in which the subset measureis being maximized, the piece of data and/or data group that would makethe largest contribution is the piece of data and/or data group havingthe largest contribution factor. In some embodiments, a value can beassociated with the piece of data outside of the subset that would makethe largest contribution to the optimization of the subset measure toidentify the piece of data and/or data group as such.

After the piece of data and/or data group that would make the largestcontribution to the optimization of the subset measure is identified,the process 1500 proceeds to block 1508 wherein the contribution factorsof the piece of data and/or data group that would make the smallestcontribution to the optimization of the subset measure and the piece ofdata and/or data group that would make the largest contribution to theoptimization of the subset measure are compared. In some embodiments,this comparison can determine whether the contribution that would bemade if the piece of data and/or data group that would make the largestcontribution to the optimization of the subset measure were included inthe subset is larger than the contribution of the piece of data and/ordata group in the subset making the smallest contribution to theoptimization of the subset measure. In embodiments in which the subsetmeasure is being maximized, this block can include determining which ofthe pieces of data has a larger contribution factor. Thus, thiscomparison can determine whether the optimization measure is moreoptimized with the current pieces of data and/or data groups than itwould be with the exchange of its weakest piece of data and/or datagroup with the strongest piece of data and/or data group that is notincluded in the subset.

In some embodiments, this comparison can be performed according to aBoolean function, wherein a first value, corresponding to a “true”condition, is associated with the subset if the subset measure would bemore optimized by replacing the weakest of the pieces of data and/ordata groups of the subset and a second value, corresponding to a “false”condition, is associated with the subset if the subset measure would notbe more optimized replacing the weakest of the pieces of data and/ordata groups of the subset.

After the contribution factors have been compared, the process 1500proceeds to decision state 1510, wherein it determines whether toexchange the identified weakest piece of data and/or data group in thesubset with the identified strongest piece of data and/or data groupoutside of the subset. In some embodiments, this determination can bemade by retrieving the Boolean-value associated with the subset, andindicative of the result of the comparison of the contribution factorsof the pieces of data. If the second value is associated with thesubset, then the pieces of data are not exchanged and the process 1500proceeds to block 1512, wherein the subset is identified as theoptimized. In some embodiments, a value can be associated with some orall of the pieces of data and/or data groups in the optimized subset.

After the subset has been identified as maximized, the process 1500proceeds to block 1514, and continues with block 1434 of FIG. 17.Returning again to decision state 1510, wherein, if the first value isassociated with the subset, then the process 1500 proceeds to block1516, and the pieces of data are exchanged. Specifically, the weakestpiece of data and/or data group in the subset is replaced with thestrongest piece of data and/or data group outside of the subset. Afterthe pieces of data have been exchanged, the process 1500 returns toblock 1504, and proceeds as outlined above. In some embodiments, theprocess 1500 can be repeated until the subset is identified asoptimized.

With reference now to FIG. 19, a flowchart illustrating an alternativeembodiment of a process 1600 for selecting a subset is shown. Theprocess 1600 can be performed by one or several components of thedistributed processing system 100, and in some embodiments, the process1600 can be performed by the server 102. In some embodiments, process1600 can be performed in connection with some or all of the steps ofprocess 1400. In the embodiment depicted in FIG. 19, process 1600 can beperformed in place of the steps of process 1400 shown in blocks 1426 to1434.

As seen in FIG. 19, process 1600 continues from block 1424 of FIG. 17and block 1601, and proceeds to block 1602, wherein the attributevectors are set to a common origin. In some embodiments, this commonorigin can be achieved by setting the values for each of the dimensionsof the vectors to a common value, and in some embodiments, by settingthe values for each of the dimensions of the vectors to zero. After thecommon origin has been set, the process 1600 proceeds to block 1604,wherein the distances between the ends of the attribute vectors aredetermined. In some embodiments, this can include calculating thedistance between the ends of some or all of the possible pairs ofvectors of the processing tasks. Further, in some embodiments, valuesidentifying a determined distance between the ends of a pair of vectorscan be associated with that pair of vectors and can be stored.

After the distance between the ends of the attribute vectors has beendetermined, the process 1600 proceeds to block 1606, wherein the vectorpair having ends separated by the greatest distance is identified. Avalue identifying the pair having ends separated by the greatestdistance can be associated with the vector pair. After the vector pairhaving ends separated by the greatest distance is identified, theprocess 1600 proceeds to block 1608 wherein the pieces of data and/ordata groups associated with the vectors in the vector pair having thegreatest distance between their ends are included in the subset. In someembodiments, this inclusion in the subset can be achieved by associatinga value indicative of inclusion in the subset with one or both of thepieces of data and/or data groups associated with the vectors formingthe vector pair. After and/or simultaneous with the inclusion of thepieces of data and/or groups of data in the subset, a count can beincremented as indicated in block 1610. In some embodiments, this countcan identify the number of pieces of data and/or data groups included inthe subset. After the count has been incremented, the process 1600proceeds to block 1612, wherein the count is compared to the desiredsize of the subset. In some embodiments, this comparison can beperformed according to a Boolean function, wherein a first value,corresponding to a “true” condition, is associated with the subset ifthe count is smaller than the desired number of pieces of data and/ordata groups in the subset and a second value, corresponding to a “false”condition, is associated with the subset if the count matches thedesired number of pieces of data and/or data groups in the subset.

After the count has been compared with the desired number of pieces ofdata and/or data groups in the subset, the process 1600 proceeds todecision state 1614, wherein it is determined whether to add additionalpieces of data and/or data groups to the subset. In some embodiments,this determination can be made by retrieving the Boolean-valueassociated with the subset, and indicative of the result of thecomparison of the count to the desired number of pieces of data and/ordata groups in the subset.

If the first value is associated with the processing task, thenadditional pieces of data and/or data groups are to be added to thesubset, and the process 1600 proceeds to block 1616, wherein thedistance between the ends of the attribute vectors in the subset and theends of the attribute vectors outside of the subset are determined. Insome embodiments, this is performed by pairing each of the vectorswithin the subset with each of the vectors outside of the subset anddetermining the distance between the ends of the vectors forming thepairs. In some embodiments, this distance can be determined by pairingeach of the vectors most recently added to the subset, and in someembodiments, each of the vectors from the vector pair most recentlyadded to the subset, with some or all of the vectors outside of thesubset. After the distances have been determined, the process 1600proceeds to block 1618, wherein one or several additional vectors areidentified. In some embodiments, these one or several additional vectorsare identified because of the additional diversity that these one orseveral vectors would add to the subset by their inclusion in thesubset. This additional diversity can be determined in a number ofdifferent ways. In some embodiments, for example, the one or severaladditional vectors can be selected as the vectors having ends separatedby the greatest distance from the ends of vectors in the subset. Inanother embodiment, these one or several additional vectors can beselected for inclusion in the subset by determining the distance betweentheir ends and the ends of the vectors in the subset. For each vector,the smallest of the distances between their end and the ends of thevectors in the subset is determined, and the one or several vectors areadded that have the greatest, smallest distance between their ends andthe ends of the vectors in the subset. After these vectors have beenidentified, the process 1600 returns to block 1608, and proceeds asoutlined above.

Returning again to decision state 1620, if the second value isassociated with the processing task, then no additional pieces of dataand/or groups of data are to be added to the subset, and the process1600 proceeds to block 1620, and returns to block 1436 of FIG. 17.

A number of variations and modifications of the disclosed embodimentscan also be used. Specific details are given in the above description toprovide a thorough understanding of the embodiments. However, it isunderstood that the embodiments may be practiced without these specificdetails. For example, well-known circuits, processes, algorithms,structures, and techniques may be shown without unnecessary detail inorder to avoid obscuring the embodiments.

Implementation of the techniques, blocks, steps and means describedabove may be done in various ways. For example, these techniques,blocks, steps and means may be implemented in hardware, software, or acombination thereof. For a hardware implementation, the processing unitsmay be implemented within one or more application specific integratedcircuits (ASICs), digital signal processors (DSPs), digital signalprocessing devices (DSPDs), programmable logic devices (PLDs), fieldprogrammable gate arrays (FPGAs), processors, controllers,micro-controllers, microprocessors, other electronic units designed toperform the functions described above, and/or a combination thereof.

Also, it is noted that the embodiments may be described as a processwhich is depicted as a flowchart, a flow diagram, a swim diagram, a dataflow diagram, a structure diagram, or a block diagram. Although adepiction may describe the operations as a sequential process, many ofthe operations can be performed in parallel or concurrently. Inaddition, the order of the operations may be re-arranged. A process isterminated when its operations are completed, but could have additionalsteps not included in the figure. A process may correspond to a method,a function, a procedure, a subroutine, a subprogram, etc. When a processcorresponds to a function, its termination corresponds to a return ofthe function to the calling function or the main function.

Furthermore, embodiments may be implemented by hardware, software,scripting languages, firmware, middleware, microcode, hardwaredescription languages, and/or any combination thereof. When implementedin software, firmware, middleware, scripting language, and/or microcode,the program code or code segments to perform the necessary tasks may bestored in a machine readable medium such as a storage medium. A codesegment or machine-executable instruction may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a script, a class, or any combination of instructions,data structures, and/or program statements. A code segment may becoupled to another code segment or a hardware circuit by passing and/orreceiving information, data, arguments, parameters, and/or memorycontents. Information, arguments, parameters, data, etc. may be passed,forwarded, or transmitted via any suitable means including memorysharing, message passing, token passing, network transmission, etc.

For a firmware and/or software implementation, the methodologies may beimplemented with modules (e.g., procedures, functions, and so on) thatperform the functions described herein. Any machine-readable mediumtangibly embodying instructions may be used in implementing themethodologies described herein. For example, software codes may bestored in a memory. Memory may be implemented within the processor orexternal to the processor. As used herein the term “memory” refers toany type of long term, short term, volatile, nonvolatile, or otherstorage medium and is not to be limited to any particular type of memoryor number of memories, or type of media upon which memory is stored.

Moreover, as disclosed herein, the term “storage medium” may representone or more memories for storing data, including read only memory (ROM),random access memory (RAM), magnetic RAM, core memory, magnetic diskstorage mediums, optical storage mediums, flash memory devices and/orother machine readable mediums for storing information. The term“machine-readable medium” includes, but is not limited to portable orfixed storage devices, optical storage devices, and/or various otherstorage mediums capable of storing that contain or carry instruction(s)and/or data.

While the principles of the disclosure have been described above inconnection with specific apparatuses and methods, it is to be clearlyunderstood that this description is made only by way of example and notas limitation on the scope of the disclosure.

What is claimed is:
 1. A system, comprising: a database coupled to acomputer network and storing: a plurality of tasks; and an accuracyparameter for each of the plurality of tasks; a first computing devicecoupled to the computer network; a second computing device coupled tothe computer network; and a server, comprising at least one serverhardware computing device coupled to the computer network and includingat least one processor executing instructions within memory which, whenexecuted, cause the system to: select, from the database, a task withinthe plurality of tasks; transmit the task through the computer networkto the first computing device and the second computing device; receive,through the computer network, a first evaluation characterizing the taskfrom the first computing device and a second evaluation characterizingthe task from the second computing device; select, from the database,the accuracy parameter for the task, the accuracy parametercharacterizing an attribute common to the first evaluation and thesecond evaluation; generate a score for the accuracy parameter accordingto at least one difference between the first evaluation and the secondevaluation; and update the accuracy parameter within the database toreflect the score.
 2. The system of claim 1, wherein the instructionsfurther cause the system to: identify a plurality of computing devicessharing a common computing device parameter; and select the firstcomputing device and the second computing device from the plurality ofcomputing devices.
 3. The system of claim 2, wherein the instructionsfurther cause the system to: transmit at least one additional evaluationthrough the computer network to at least one additional computing devicein the plurality of computing devices; receive, from the at least oneadditional computing device, at least one additional evaluation;generate the score according to at least one difference between thefirst evaluation, the second evaluation, and the at least one additionalevaluation; and update the accuracy parameter within the database toreflect the score.
 4. The system of claim 3, wherein the instructionsfurther cause the system to coordinate a plurality of processingresources in the plurality of computing devices to complete the task. 5.The system of claim 4, wherein the instructions further cause the systemto: transmit a first instruction to the first computing device tocomplete a first portion of the task; and transmit a second instructionto the second computing device to complete a second portion of the task.6. The system of claim 4, wherein the instructions further cause thesystem to: identify, within the plurality of computing devices, aplurality of latent or unused resources; and coordinate the plurality oflatent or unused resources to complete at least one additional task. 7.The system of claim 1, wherein the instructions further cause the systemto: divide the task into a plurality of subtasks; generate, from each ofthe plurality of subtasks, a metric indicating a processing efficiencyof each of the plurality of subtasks; identify, within the plurality ofsubtasks, at least one subtask associated with the metric below apredetermined threshold; and replace each of the at least one subtaskwith a replacement subtask, having the metric above the predeterminedthreshold.
 8. The system of claim 7, wherein the metric identifies atleast one contribution factor within each of the plurality of subtasks,wherein the at least one contribution factor for the at least onesubtask indicates a lower positive effect on the processing efficiency,and the at least one contribution factor for the replacement subtaskindicates a higher positive effect on the processing efficiency.
 9. Amethod comprising: storing, by a server comprising at least one serverhardware computing device coupled to a computer network and including atleast one processor executing instructions within memory, within adatabase coupled to the computer network: a plurality of tasks; and anaccuracy parameter for each of the plurality of tasks; selecting, by theserver from the database, a task within a plurality of tasks;transmitting, by the server, the task through the computer network to afirst computing device coupled to the computer network, and to a secondcomputing device coupled to the computer network; receiving, by theserver through the computer network, a first evaluation characterizingthe task from the first computing device and a second evaluationcharacterizing the task from the second computing device; selecting, bythe server from the database, the accuracy parameter for the task, theaccuracy parameter characterizing an attribute common to the firstevaluation and the second evaluation; generating, by the server, a scorefor the accuracy parameter according to at least one difference betweenthe first evaluation and the second evaluation; and updating, by theserver, the accuracy parameter within the database to reflect the score.10. The method of claim 9, further comprising the step of coordinatingthe completion of the task, as a processing task, among a plurality ofprocessing resources within a plurality of computing devices includingthe first computing device and the second computing device.
 11. Themethod of claim 9, wherein the first computing device is operated in adistributed processing environment in a first geographical location, andthe second computing device is operated in the distributed processingenvironment in a second geographical location.
 12. The method of claim9, further comprising the step of receiving: the first evaluation as afirst user input from the first computing device and the secondevaluation as a second user input from the second computing device; orthe first evaluation as an output from a first automated process on thefirst computing device and the second evaluation as a second automatedprocess on the second computing device.
 13. The method of claim 9,further comprising the steps of: dividing, by the server, the task intoa plurality of subtasks; generating, by the server from each of theplurality of subtasks, a metric indicating a processing efficiency ofeach of the plurality of subtasks; identifying, by the server within theplurality of subtasks, at least one subtask associated with the metricbelow a predetermined threshold; and replacing, by the server, each ofthe at least one subtask with a replacement subtask, having the metricabove the predetermined threshold.
 14. The method of claim 13, whereinthe metric identifies at least one contribution factor within each ofthe plurality of subtasks, wherein the at least one contribution factorfor the at least one subtask indicates a lower positive effect on theprocessing efficiency, and the at least one contribution factor for thereplacement subtask indicates a higher positive effect on the processingefficiency.
 15. A system comprising a server, comprising at least oneserver hardware computing device coupled to a computer network andincluding at least one processor executing instructions within memory,configured to: store, within a database coupled to the computer network:a plurality of tasks; and an accuracy parameter for each of theplurality of tasks; select, from the database, a task within a pluralityof tasks; transmit the task through the computer network to a firstcomputing device coupled to the computer network, and to a secondcomputing device coupled to the computer network; receive, through thecomputer network, a first evaluation characterizing the task from thefirst computing device and a second evaluation characterizing the taskfrom the second computing device; select, from the database, theaccuracy parameter for the task, the accuracy parameter characterizingan attribute common to the first evaluation and the second evaluation;generate a score for the accuracy parameter according to at least onedifference between the first evaluation and the second evaluation; andupdate the accuracy parameter within the database to reflect the score.16. The system of claim 15, wherein the server is further configured to:divide the task into a plurality of subtasks; generate, from each of theplurality of subtasks, a metric indicating a processing efficiency ofeach of the plurality of subtasks; identify, within the plurality ofsubtasks, at least one subtask associated with the metric below apredetermined threshold; and replace each of the at least one subtaskwith a replacement subtask, having the metric above the predeterminedthreshold.
 17. The system of claim 16, wherein the metric identifies atleast one contribution factor within each of the plurality of subtasks,wherein the at least one contribution factor for the at least onesubtask indicates a lower positive effect on the processing efficiency,and the at least one contribution factor for the replacement subtaskindicates a higher positive effect on the processing efficiency.
 18. Thesystem of claim 15, wherein the server is further configured tocoordinate a plurality of processing resources in a plurality ofcomputing devices to complete the task.
 19. The system of claim 18,wherein the server is further configured to: transmit a firstinstruction to the first computing device to complete a first portion ofthe task; and transmit a second instruction to the second computingdevice to complete a second portion of the task.
 20. The system of claim18, wherein the server is further configured to: identify, within theplurality of computing devices, a plurality of latent or unusedresources; and coordinate the plurality of latent or unused resources tocomplete at least one additional task.